Image processing method, electronic device, and computer readable medium

Abstract

Embodiments of the present application provide an image processing method, an electronic device and a computer readable medium to solve the problem of long time consumption of image retrieval target image. The image processing method obtains a first feature map and a second feature map of a query image. And a third feature map and a fourth feature map of a sample image are obtained. The first feature map, the second feature map, the third feature map and the fourth feature map are all two-dimensional features. The first similarity result, the second similarity result, the third similarity result and the fourth similarity result are calculated by crossing the first feature map, the second feature map, the third feature map and the fourth feature map respectively, and are calculated by feature vectors. The first similarity feature map is obtained, and the similarity between the query image and the sample image is obtained. The image processing method provided by the embodiments of the present application can reduce the calculation amount and improve the speed of image retrieval.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, electronic device, and computer-readable medium. Background Technology

[0002] Image retrieval technology enables the search for "target images" similar to a "given query image" within a massive image library. Image retrieval can be performed by searching for images within the same image. The image library contains multiple sample images. For each "given query image," the similarity between the query image and each sample image in the library is calculated. The higher the similarity value, the more similar the corresponding query image and sample image are. Sample images that meet preset similarity criteria can be considered "target images."

[0003] With the development of information technology and the internet, the number of sample images in image libraries has grown rapidly. The hundreds of millions of sample images make image retrieval methods using image search time-consuming. How to quickly retrieve target images from massive image libraries has become an urgent problem to be solved. Summary of the Invention

[0004] This application provides an image processing method, an electronic device, and a computer-readable medium to solve the problem of long image retrieval time for target images.

[0005] In a first aspect, an image processing method is provided, comprising: determining a first feature map of a query image, the first feature map including two-dimensional features of the query image; obtaining a second feature map based on the first feature map; the first feature map and the second feature map having different sizes; determining a third feature map of a sample image, the third feature map including two-dimensional features of the sample image; obtaining a fourth feature map based on the third feature map; the third feature map and the fourth feature map having different sizes; obtaining a first similarity result based on the first feature map and the third feature map; obtaining a second similarity result based on the first feature map and the fourth feature map; obtaining a third similarity result based on the second feature map and the third feature map; obtaining a fourth similarity result based on the second feature map and the fourth feature map; and obtaining a first similarity feature map based on the first similarity result, the second similarity result, the third similarity result, and the fourth similarity result.

[0006] Based on the image processing method provided in the above embodiments of this application, it can be seen that the image processing method obtains two two-dimensional feature maps of different sizes for the query image, namely a first feature map and a second feature map. The first and second feature maps are used to characterize different vector features of the query image. Also, it obtains two two-dimensional feature maps of different sizes for the sample image, namely a third feature map and a fourth feature map. The third and fourth feature maps are used to characterize different vector features of the sample image. The first, second, third, and fourth feature maps are all two-dimensional features. When calculating similarity, the first similarity result is obtained based on the first and third feature maps. That is, similarity is calculated using feature vectors. Similarly, the second similarity result is obtained based on the first and fourth feature maps. The third similarity result is obtained based on the second and third feature maps. The fourth similarity result is obtained based on the second and fourth feature maps. All of the above are calculated using feature vectors, and then the first similarity feature map is obtained based on the first, second, third, and fourth similarity results, thereby obtaining the similarity between the query image and the sample image. Compared to directly querying the original image and calculating similarity, the image processing method provided in this application can reduce the amount of computation and improve the speed of image retrieval.

[0007] Furthermore, in the embodiments of this application, the first similarity result, the second similarity result, the third similarity result, and the fourth similarity result are obtained by cross-calculation of the first feature map, the second feature map, and the third feature map and the fourth feature map, respectively. This enables the operation of matching the local features of the query map with the local features of the target map, and the first similarity feature map is more accurate in representing the similarity between the query map and the sample map. This is beneficial for finding a sample map that is more similar to the query map, making the results of image search more accurate.

[0008] In a feasible implementation of the first aspect, the image processing method further includes: determining multiple two-dimensional feature maps of different sizes for the query image; determining multiple two-dimensional feature maps of different sizes for the sample image; and cross-calculating each two-dimensional feature map of the query image with each two-dimensional feature map of the sample image to obtain a first similarity feature map.

[0009] In this way, cross-computation can complete the operation of matching the local features of the query graph with the local features of the target graph, which is beneficial for finding sample graphs that are more similar to the query graph, making the results of graph search more accurate.

[0010] In a feasible implementation of the first aspect, the image processing method further includes: obtaining a fifth feature map based on the first feature map; the size of the first feature map is larger than that of the second feature map, and the size of the second feature map is larger than that of the fifth feature map; obtaining a fifth similarity result based on the fifth feature map and the third feature map; obtaining a sixth similarity result based on the fifth feature map and the fourth feature map; and obtaining a first similarity feature map based on the fifth similarity result and the sixth similarity result.

[0011] By obtaining the fifth feature map, a two-dimensional feature map of the query image is added. The fifth and sixth similarity results enrich the first similarity feature map, further increasing the accuracy of image search results.

[0012] In a feasible implementation of the first aspect, the image processing method further includes: obtaining a sixth feature map based on a third feature map; the size of the third feature map is larger than that of the fourth feature map, and the size of the fourth feature map is larger than that of the sixth feature map; obtaining a seventh similarity result based on the first feature map and the sixth feature map; obtaining an eighth similarity result based on the second feature map and the sixth feature map; obtaining a ninth similarity result based on the fifth feature map and the sixth feature map; and obtaining a first similarity feature map based on the seventh similarity result, the eighth similarity result, and the ninth similarity result.

[0013] By obtaining the sixth feature map, the two-dimensional feature map of the sample image is increased. The seventh, eighth, and ninth similarity results can enrich the first similarity feature map, further increasing the accuracy of image search results.

[0014] In a feasible implementation of the first aspect, the image processing method further includes: determining a spatial attention map of the query image; using the spatial attention map to characterize key regions of the image; and obtaining the similarity between the query image and the sample image based on the first similarity feature map and the spatial attention map.

[0015] In this way, by calculating a spatial attention map using the query map and applying it to the first similarity feature map, the similarity between the query map and the sample map is calculated, thus improving the accuracy of the similarity calculation. For example, the image processing method provided in this application embodiment can eliminate the influence of large areas of irrelevant background on the similarity result calculation.

[0016] In a feasible implementation of the first aspect, the similarity between the query image and the sample image is obtained based on the first similarity feature map and the spatial attention map, including: combining the first similarity feature map and the spatial attention map through a convolutional neural network; wherein the convolutional neural network includes multiple different convolutional layers; the first similarity feature map is passed through one or more convolutional layers to obtain a first convolutional feature map; the first convolutional feature map is multiplied by the spatial attention map to obtain a first encoding map; the first encoding map is passed through one or more layers in sequence to obtain a second similarity feature map; and the second similarity feature map is globally averaged and normalized to obtain the similarity.

[0017] One or more convolutional layers can further extract features and perform in-depth processing, which helps to increase the accuracy of image search results.

[0018] In a feasible implementation of the first aspect, the similarity between the query image and the sample image is obtained based on the first similarity feature map and the spatial attention map, including: combining the first similarity feature map and the spatial attention map through a convolutional neural network; wherein the convolutional neural network includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, and a fifth convolutional layer; the first similarity feature map is passed through the first convolutional layer and the second convolutional layer respectively to obtain a first convolutional feature map; the first convolutional feature map is multiplied by the spatial attention map to obtain a first encoding map; the first encoding map is passed through the third convolutional layer, the fourth convolutional layer, and the fifth convolutional layer in sequence to obtain a second similarity feature map; the second similarity feature map is globally averaged and normalized to obtain the similarity. The first convolutional layer, the second convolutional layer, the third convolutional layer, the fourth convolutional layer, and the fifth convolutional layer are different convolutional layers, for example, the stride of the convolutional kernel is different, and / or the size of the convolutional kernel is different.

[0019] Thus, the image processing method provided in this application embodiment can obtain a spatial attention-weighted feature map, namely the second similarity feature map. When calculating similarity using the second similarity feature map, more attention is paid to the similarity of key regions in the image, further increasing the accuracy of image search results. For example, when both the query image and the sample image include large areas of white background, this application embodiment can eliminate the interference of large areas of white background.

[0020] Furthermore, the first encoding map is passed through the third, fourth, and fifth convolutional layers in sequence to obtain the second similarity feature map, which can further extract features and help increase the accuracy of image search results.

[0021] In the feasible implementation of the first aspect, the size of the first feature map is the same as the size of the third feature map; the size of the second feature map is the same as the size of the fourth feature map.

[0022] In a feasible implementation of the first aspect, obtaining a second feature map based on a first feature map includes: obtaining a second feature map from a first feature map through convolution and shrinking.

[0023] The image processing method provided in this application calculates image similarity through two-dimensional convolution. Two-dimensional convolution has stronger interpretability and is conducive to achieving fast and accurate image retrieval.

[0024] In the feasible implementation of the first aspect, the spatial attention map of the query graph is determined, including: obtaining the average and maximum values ​​of the two-dimensional features of the query graph in the feature dimension; and concatenating the average and maximum value feature dimensions to obtain the spatial attention map.

[0025] In the feasible implementation of the first aspect, the spatial attention map of the query graph is determined, including: obtaining the average and maximum values ​​of the two-dimensional features of the query graph in the feature dimension; concatenating and normalizing the average and maximum value feature dimensions to obtain the spatial attention map.

[0026] In a feasible implementation of the first aspect, a first similarity feature map is obtained based on the first similarity result, the second similarity result, the third similarity result, and the fourth similarity result, including: enlarging the first similarity result, the second similarity result, the third similarity result, and the fourth similarity result to make the first similarity result, the second similarity result, the third similarity result, and the fourth similarity result the same size; and splicing the first similarity result, the second similarity result, the third similarity result, and the fourth similarity result.

[0027] In a second aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the image processing method provided in the first aspect.

[0028] The electronic device provided in this application embodiment can execute the image processing method as provided in the first aspect. In this way, the performance of the electronic device in various applications involving image search can be improved, thereby improving the overall performance of the electronic device and enabling the electronic device to adapt to more application scenarios.

[0029] Thirdly, a computer-readable storage medium is provided having computer program instructions stored thereon, which can be executed by a processor to implement the image processing method as provided in the first aspect.

[0030] The computer-readable storage medium provided in this application embodiment stores computer program instructions capable of executing the image processing method provided in the first aspect. Thus, the computer-readable storage medium can achieve a dual improvement in the speed and accuracy of image search. Attached Figure Description

[0031] Figure 1 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0032] Figure 2 A schematic flowchart illustrating an image processing method provided in another embodiment of this application;

[0033] Figure 3 A schematic flowchart of an image processing method provided in another embodiment of this application;

[0034] Figure 4 A schematic flowchart of an image processing method provided in another embodiment of this application;

[0035] Figure 5 A schematic flowchart of an image processing method provided in another embodiment of this application;

[0036] Figure 6 A schematic diagram of a query diagram provided in an embodiment of this application;

[0037] Figure 7 A schematic diagram of a sample image provided for an embodiment of this application;

[0038] Figure 8 A schematic diagram of the query graph and spatial attention graph provided in the embodiments of this application;

[0039] Figure 9 This is a schematic diagram of the structure of a device suitable for implementing the solutions in the embodiments of this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0041] This application provides an image processing method that can be used to implement image retrieval (IR) in computer vision (CV). Computer vision is an important field within computer science and artificial intelligence, researching the use of computers to recognize, process, and generate digital images. Image retrieval is a crucial topic in computer vision, studying the retrieval of target images from massive image databases using text, images, and other methods. The image retrieval method described in this application refers to retrieving target images using a query graph.

[0042] In some applications, image retrieval includes image retrieval re-ranking. When performing image retrieval, most images with low similarity can be filtered out first, resulting in a small number of candidate samples with higher similarity. These candidate samples are then used to recalculate the similarity between the query image and the target image using a re-ranking model, and the retrieval results are re-ranked based on the new similarity. This results in a more accurate similarity assessment between the target image and the query image.

[0043] The following describes the image processing method provided in the embodiments of this application by applying the image processing method to image retrieval and rearrangement.

[0044] To find sample images similar to the query image more quickly and accurately, in some embodiments, multiple two-dimensional feature maps of different sizes for the query image are determined. Multiple two-dimensional feature maps of different sizes for the sample image are also determined. Each two-dimensional feature map of the query image is cross-calculated with each two-dimensional feature map of the sample image to obtain a first similarity feature map. This cross-calculation enables the matching of local features of the query image with local features of the target image, which is beneficial for finding sample images more similar to the query image, making the image search results more accurate.

[0045] The number of two-dimensional feature maps of different sizes in the query image can be two, three, or four. This application does not limit the number of two-dimensional feature maps of different sizes in the query image. Similarly, the number of two-dimensional feature maps of different sizes in the sample image can be two, three, or four. This application does not limit the number of two-dimensional feature maps of different sizes in the sample image.

[0046] The image processing method provided in this application embodiment will be described below using two two-dimensional feature maps of different sizes of the query image and two two-dimensional feature maps of different sizes of the sample image as examples.

[0047] Figure 1 This is a schematic flowchart illustrating an image processing method provided in an embodiment of this application. Figure 1 As shown, the image processing method includes the following processing steps:

[0048] Step S101: Determine the first feature map of the query graph, the first feature map including the two-dimensional features of the query graph.

[0049] For example, the size of the first feature map of the query graph can be 40×40.

[0050] Step S102: Obtain the second feature map based on the first feature map; the first feature map and the second feature map are of different sizes.

[0051] In one implementation, the first feature map can be convolved and scaled down to obtain the second feature map. The image processing method provided in this application calculates image similarity through two-dimensional convolution. Two-dimensional convolution has stronger interpretability and is beneficial for fast and accurate image retrieval. For example, when the size of the first feature map of the query image is 40×40, the size of the second feature map of the query image can be 20×20.

[0052] Step S103: Determine the third feature map of the sample image. The third feature map includes the two-dimensional features of the sample image.

[0053] In one implementation, the size of the first feature map of the query graph can be the same as the size of the third feature map of the sample graph. For example, the size of the third feature map of the sample graph can be 40×40.

[0054] In another implementation, steps S101 and S103 can be performed simultaneously. In yet another implementation, step S103 can be executed before step S101. This application does not impose any limitations.

[0055] Step S104: Obtain the fourth feature map based on the third feature map; the third feature map and the fourth feature map are of different sizes.

[0056] In one implementation, the third feature map can be convolved and scaled down to obtain the fourth feature map. The image processing method provided in this application calculates image similarity through two-dimensional convolution. Two-dimensional convolution has stronger interpretability, which is beneficial for fast and accurate image retrieval. For example, when the size of the third feature map of the sample image is 40×40, the size of the fourth feature map of the sample image can be 20×20.

[0057] Step S105: Obtain the first similarity result based on the first feature map and the third feature map.

[0058] In one implementation, the features of the first feature map Q are Q∈R. D×H1×W1 Where D represents the feature dimension, H1 represents the feature map height, and W1 represents the feature map width. The features of the third feature map G ∈ R. D×H2×W2 Where D represents the feature dimension, H2 represents the feature map height, and W2 represents the feature map width. The first similarity result S can be calculated as S∈R (H2*W2)×H1×W1 For example, given a D-dimensional feature F_ij at any position of Q, where i represents the index of the feature map height dimension and j represents the index of the feature map width dimension, the cosine similarity between F_ij and the positional features of G is calculated and expanded into a high-dimensional vector of (H2*W2) dimensions, which is S_ij. This results in a shorter computation time for the first similarity result.

[0059] Step S106: Obtain the second similarity result based on the first feature map and the fourth feature map.

[0060] This step can be found in step S105, and will not be repeated here.

[0061] In one implementation, the third and fourth feature maps are of different sizes, and therefore the sizes of the first and second similarity results are also different.

[0062] Step S107: Based on the second feature map and the third feature map, obtain the third similarity result.

[0063] This step can be found in step S105, and will not be repeated here.

[0064] Step S108: Based on the second feature map and the fourth feature map, obtain the fourth similarity result.

[0065] This step can be found in step S105, and will not be repeated here.

[0066] In one implementation, the third and fourth feature maps are of different sizes, and therefore the third and fourth similarity results are also of different sizes.

[0067] Step S109: Based on the first similarity result, the second similarity result, the third similarity result, and the fourth similarity result, obtain the first similarity feature map.

[0068] In one implementation, the sizes of the first, second, third, and fourth similarity results vary. The first, second, third, and fourth similarity results are enlarged to make them all the same size. The first, second, third, and fourth similarity results are then concatenated. For example, the largest similarity result among the first, second, third, and fourth similarity results is used as a benchmark, and these results are then enlarged. That is, the enlargement factor of one of the similarity results can be one.

[0069] The first, second, third, and fourth feature maps are all two-dimensional features. When calculating similarity, the first similarity result is obtained based on the first and third feature maps. That is, similarity is calculated using feature vectors. Similarly, the second, third, and fourth similarity results are also calculated using feature vectors. Compared to directly calculating similarity using the original image of the query image, the image processing method provided in this application reduces computational load and improves image retrieval speed.

[0070] Furthermore, in the embodiments of this application, the first similarity result, the second similarity result, the third similarity result, and the fourth similarity result are obtained by cross-calculation of the first feature map, the second feature map, and the third feature map and the fourth feature map, respectively. This enables the operation of matching the local features of the query map with the local features of the target map, and the first similarity feature map is more accurate in representing the similarity between the query map and the sample map. This is beneficial for finding a sample map that is more similar to the query map, making the results of image search more accurate.

[0071] This application embodiment calculates image similarity using two-dimensional (2D) convolution. 2D convolution has stronger interpretability, and the connection between it and the query image or sample image is more intuitive. Compared to the lack of interpretability in 4D convolution, this application embodiment can adapt to more application scenarios using 2D convolution.

[0072] To calculate similarity more accurately, three or more two-dimensional feature maps of different sizes from the query image can be used to calculate the similarity results with three or more two-dimensional feature maps of different sizes from the sample image, thus enriching the first similarity feature map. The following explanation uses three two-dimensional feature maps as an example.

[0073] The query graph has three 2D feature maps of different sizes. At this point, such as... Figure 2 As shown, steps S201 to S204 are executed.

[0074] Step S201: Obtain the fifth feature map based on the first feature map. The size of the first feature map is larger than that of the second feature map, and the size of the second feature map is larger than that of the fifth feature map.

[0075] For example, if the size of the first feature map of the query graph is 40×40 and the size of the second feature map of the query graph is 20×20, the size of the fifth feature map of the query graph can be 10×10.

[0076] In one implementation, the fifth feature map of the query map can be obtained by convolution and shrinking based on the second feature map of the query map.

[0077] In one implementation, step S201 is executed after step S102 and before step S109.

[0078] Step S202: Based on the fifth feature map and the third feature map, obtain the fifth similarity result.

[0079] Step S203: Based on the fifth feature map and the fourth feature map, obtain the sixth similarity result.

[0080] In some embodiments, steps S201 to S203 are performed before step S109.

[0081] Step S204: Based on the fifth similarity result and the sixth similarity result, obtain the first similarity feature map.

[0082] In some embodiments, step S204 is performed simultaneously with step S109.

[0083] Three two-dimensional feature maps of different sizes are also set for the sample image. At this point, such as... Figure 3 As shown, steps S301 to S305 are executed.

[0084] Step S301: Obtain the sixth feature map based on the third feature map. The size of the third feature map is larger than that of the fourth feature map, and the size of the fourth feature map is larger than that of the sixth feature map.

[0085] Step S302: Based on the first feature map and the sixth feature map, obtain the seventh similarity result.

[0086] Step S303: Based on the second feature map and the sixth feature map, the eighth similarity result is obtained.

[0087] Step S304: Based on the fifth and sixth feature maps, obtain the ninth similarity result.

[0088] In some embodiments, steps S302 to S304 are performed before step S109. By performing steps S302 to S304, cross-calculation is performed on the two-dimensional feature map of the query graph and the two-dimensional feature map of the sample graph.

[0089] Step S305: Based on the seventh similarity result, the eighth similarity result, and the ninth similarity result, the first similarity feature map is obtained.

[0090] In some embodiments, step S305 is executed concurrently with step S109. In one implementation method, a first similarity feature map is obtained by concatenating the first similarity result, the second similarity result, the third similarity result, the fourth similarity result, the fifth similarity result, the sixth similarity result, the seventh similarity result, the eighth similarity result, and the ninth similarity result. For example, the size of the first similarity result is f1*40*40, where f1 represents the feature dimension of the first similarity result, and 40*40 represents the spatial dimension of the first similarity result. The size of the second similarity result is f2*20*20, where f2 represents the feature dimension of the second similarity result, and 20*20 represents the spatial dimension of the second similarity result. The size of the second similarity result is enlarged to f2*40*40. The size of the third similarity result is f3*10*10, where f3 represents the feature dimension of the third similarity result, and 10*10 represents the spatial dimension of the third similarity result. The size of the third similarity result is enlarged to f3*40*40. Similarly, the size of the fifth similarity result is enlarged to f5*40*40. The size of the sixth similarity result is enlarged to f6*40*40. The size of the seventh similarity result is enlarged to f7*40*40. The size of the eighth similarity result is enlarged to f8*40*40. The size of the ninth similarity result is enlarged to f9*40*40. The final first similarity feature map, concatenated by feature dimensions, is (f1+f2+f3…+f9)*40*40.

[0091] By obtaining the sixth feature map, the two-dimensional feature map of the sample image is increased. The seventh, eighth, and ninth similarity results can enrich the first similarity feature map, further increasing the accuracy of image search results.

[0092] It is understood that the above embodiments only exemplify the scenario where the query graph has three two-dimensional feature maps and the sample graph has three two-dimensional feature maps, and do not constitute a limitation on the number of two-dimensional maps. For example, the query graph can have four two-dimensional feature maps of different sizes, and the sample graph can also have four two-dimensional feature maps of different sizes.

[0093] The above operation of matching the local features of the query image with the local features of the target image to obtain the first similarity feature map is more accurate in representing the similarity between the query image and the sample image. This is beneficial for finding a sample image that is more similar to the query image, making the results of image search more accurate.

[0094] In some scenarios, query and sample images include backgrounds that occupy a large area of ​​the image. When the background is a non-critical area of ​​the image, its large size can interfere with the similarity results. For example, ... Figure 6 The query graph shown and such Figure 7 The sample images shown all include large areas of white background. Figure 6 The key area of ​​the query graph shown is the cat, while... Figure 7 The key area of ​​the sample image shown is people.

[0095] To eliminate interference from non-critical areas, this embodiment of the application executes steps S101 to S109. For example... Figure 4 As shown, the image processing method also includes:

[0096] Step S401: Determine the spatial attention map of the query graph. The spatial attention map is used to represent the key regions of the image.

[0097] Spatial attention is an attention mechanism. It calculates a two-dimensional attention map using a two-dimensional feature map. The value of this attention map ranges from 0 to 1. A higher value indicates a more important region. For example, when the attention map value reaches a certain preset condition, the corresponding region is considered a priority region.

[0098] In one implementation, the average and maximum values ​​of the two-dimensional features of the query graph are obtained along the feature dimension. The average and maximum value feature dimensions are then concatenated and normalized to obtain the spatial attention map.

[0099] For example, the two-dimensional feature M of the query graph is represented as M∈R K×H×W Where K represents the feature dimension, H represents the feature map height, and W represents the feature map width. The average value of the two-dimensional feature M of the query graph is denoted as Me, where Me ∈ R. 1×H×w The maximum value of the two-dimensional feature M of the query graph is denoted as Ma, where Ma∈R. 1×H×W .

[0100] The spatial attention map (sa) is calculated as follows:

[0101] sa=sigmoid(conv([Me, Ma]));

[0102] Where [Me, Ma] represents the concatenation of Me and Ma along the feature dimension; conv represents convolution; and normalization (sigmoid) is the sigmoid function.

[0103] Step S402: Based on the first similarity feature map and the spatial attention map, obtain the similarity between the query map and the sample map.

[0104] In one implementation, the similarity between the query image and the sample image is obtained based on a first similarity feature map and a spatial attention map, including: combining the first similarity feature map and the spatial attention map using a convolutional neural network. The convolutional neural network includes multiple convolutional layers. The first similarity feature map is passed through one or more convolutional layers to obtain a first convolutional feature map. The first convolutional feature map is multiplied by the spatial attention map to obtain a first encoded map. The first encoded map is then passed through one or more convolutional layers to obtain a second similarity feature map. The second similarity feature map is then globally averaged and normalized to obtain the similarity score. One or more convolutional layers can further extract features and perform in-depth processing, which helps to increase the accuracy of image search results.

[0105] For example, a convolutional neural network includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, and a fifth convolutional layer. Figure 5 As shown, it includes the following steps:

[0106] Step S4021: The first similarity feature map is passed through the first convolutional layer and the second convolutional layer to obtain the first convolutional feature map. The first convolutional layer and the second convolutional layer are different convolutional layers, for example, the stride of the convolutional kernel is different, and / or the size of the convolutional kernel is different.

[0107] Step S4022: Multiply the first convolutional feature map by the spatial attention map to obtain the first encoding map.

[0108] In step S4023, the first encoded map is passed sequentially through the third, fourth, and fifth convolutional layers to obtain the second similarity feature map. The third, fourth, and fifth convolutional layers are different convolutional layers, for example, the stride of the convolutional kernel is different, and / or the size of the convolutional kernel is different.

[0109] The first encoding map is passed through the third, fourth, and fifth convolutional layers in sequence to obtain the second similarity feature map, which can further extract features and help increase the accuracy of image search results.

[0110] It is understood that step S4023 is merely an example illustrating that the first encoding map passes through three convolutional layers. It does not constitute a limitation on the number of convolutional layers the first encoding map passes through. For example, in other embodiments, the first encoding map may pass through one, two, or four convolutional layers.

[0111] In some embodiments, step S4024 involves globally averaging and normalizing the second similarity feature map (sigmoid) to obtain the similarity.

[0112] In this way, by calculating a spatial attention map using the query map and applying it to the first similarity feature map, the similarity between the query map and the sample map is calculated, thus improving the accuracy of the similarity calculation. The image processing method provided in this application can obtain a spatial attention-weighted feature map, namely the second similarity feature map. When calculating similarity using the second similarity feature map, more attention is paid to the similarity of key regions in the image, further increasing the accuracy of image search results.

[0113] In some embodiments, similarity can be a value from 0 to 1, where a higher value indicates that the query graph and the sample graph are more similar.

[0114] To illustrate the accuracy of the image processing method provided in this application in calculating the similarity between query images and sample images, a "training set of 20,000 query image-candidate image pairs" and a "test set of 2,000 query image-candidate image pairs" are constructed.

[0115] Table 1 shows the accuracy of different testing methods.

[0116] Table 1

[0117] Vector retrieval 82.0 4D Convolutional Rearrangement 98.6 2D Convolutional Rearrangement (Example of this application) 98.5 2D Convolutional Rearrangement + Spatial Attention (Example from this application) 98.9

[0118] As shown in Table 1, the existing vector retrieval methods have the lowest accuracy and lag significantly behind the reordering methods (including 4D convolutional reordering and 2D convolutional reordering). It is understandable that the reordering methods refer to the image retrieval reordering mentioned earlier.

[0119] The image processing method provided in this application, by incorporating spatial attention, focuses more on key areas of the image, significantly improving accuracy. For example... Figure 8 As shown, when portrait images are used as query images, the model pays more attention to the face and hands, while the background receives a lower attention score, thus improving the accuracy of similarity.

[0120] Based on the same inventive concept, this application also provides an electronic device. The method corresponding to the electronic device can be the image processing method for image retrieval in the foregoing embodiments, and its problem-solving principle is similar to that method. The electronic device provided in this application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the methods and / or technical solutions of the foregoing embodiments of this application.

[0121] The electronic device can be a user device, or a device formed by integrating user devices and network devices through a network, or it can be an application running on the aforementioned devices. The user device includes, but is not limited to, various terminal devices such as computers, mobile phones, tablets, smartwatches, and wristbands. The network device includes, but is not limited to, network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, and can be used to implement some processing functions when setting an alarm clock. Here, the cloud consists of a large number of hosts or network servers based on cloud computing. Cloud computing is a type of distributed computing, consisting of a virtual computer composed of a group of loosely coupled computer sets.

[0122] Figure 9 The structure of an apparatus suitable for implementing the methods and / or technical solutions in the embodiments of this application is shown. For example... Figure 9 As shown, in some embodiments, the device 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1202 or a program loaded from storage portion 1208 into random access memory (RAM) 1203. The RAM 1203 also stores various programs and data required for system operation. The CPU 1201, ROM 1202, and RAM 1203 are interconnected via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0123] The following components are connected to I / O interface 1205: an input section 1206 including a keyboard, mouse, touchscreen, microphone, infrared sensor, etc.; an output section 1207 including a cathode ray tube (CRT), liquid crystal display (LCD), LED display, OLED display, etc., and speakers, etc.; a storage section 1208 including one or more computer-readable media such as hard disk, optical disk, magnetic disk, semiconductor memory, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1209 performs communication processing via a network such as the Internet.

[0124] In particular, the methods and / or embodiments in this application can be implemented as computer software programs. For example, the embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 1201, it performs the functions defined in the methods of this application.

[0125] Another embodiment of this application provides a computer-readable storage medium having computer program instructions stored thereon, which can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of this application described above.

[0126] Specifically, this embodiment may employ any combination of one or more computer-readable media. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0127] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0128] The program code contained on a computer-readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0129] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0130] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0131] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0132] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or page components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0133] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0134] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0135] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

[0137] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a device claim may also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

Claims

1. An image processing method, wherein, The image processing method includes: A first feature map of the query graph is determined, the first feature map including two-dimensional features of the query graph; a second feature map is obtained based on the first feature map; the first feature map and the second feature map are of different sizes; A third feature map of the sample image is determined, the third feature map including two-dimensional features of the sample image; a fourth feature map is obtained based on the third feature map; the third feature map and the fourth feature map are of different sizes; Based on the first feature map and the third feature map, a first similarity result is obtained; Based on the first feature map and the fourth feature map, a second similarity result is obtained; Based on the second feature map and the third feature map, a third similarity result is obtained; Based on the second feature map and the fourth feature map, a fourth similarity result is obtained; Based on the first similarity result, the second similarity result, the third similarity result, and the fourth similarity result, a first similarity feature map is obtained; Determine the spatial attention map of the query graph; the spatial attention map is used to characterize the key regions of the image; The first similarity feature map and the spatial attention map are combined using a convolutional neural network; wherein the convolutional neural network includes multiple convolutional layers; the first similarity feature map is passed through one or more of the convolutional layers to obtain a first convolutional feature map; the first convolutional feature map is multiplied by the spatial attention map to obtain a first encoding map; the first encoding map is passed through one or more of the convolutional layers to obtain a second similarity feature map; the second similarity feature map is globally averaged and normalized to obtain the similarity between the sample map and the query map.

2. The image processing method according to claim 1, characterized in that, The image processing method further includes: A fifth feature map is obtained based on the first feature map; the size of the first feature map is larger than that of the second feature map, and the size of the second feature map is larger than that of the fifth feature map; Based on the fifth feature map and the third feature map, the fifth similarity result is obtained; Based on the fifth feature map and the fourth feature map, the sixth similarity result is obtained; The first similarity feature map is obtained based on the fifth similarity result and the sixth similarity result.

3. The image processing method according to claim 2, characterized in that, The image processing method further includes: A sixth feature map is obtained based on the third feature map; the size of the third feature map is larger than that of the fourth feature map, and the size of the fourth feature map is larger than that of the sixth feature map; Based on the first feature map and the sixth feature map, a seventh similarity result is obtained; Based on the second feature map and the sixth feature map, the eighth similarity result is obtained; Based on the fifth feature map and the sixth feature map, the ninth similarity result is obtained; The first similarity feature map is obtained based on the seventh similarity result, the eighth similarity result, and the ninth similarity result.

4. The image processing method according to claim 1, characterized in that, The determination of the spatial attention graph of the query graph includes: In terms of features, the average and maximum values ​​of the two-dimensional features of the query graph are obtained; The spatial attention map is obtained by concatenating the feature dimensions of the average value and the maximum value.

5. The image processing method according to any one of claims 1-3, characterized in that, The step of obtaining the second feature map based on the first feature map includes: The second feature map is obtained by convolution and shrinking the first feature map.

6. The image processing method according to any one of claims 1-3, characterized in that, The step of obtaining the first similarity feature map based on the first similarity result, the second similarity result, the third similarity result, and the fourth similarity result includes: Enlarge the first similarity result, the second similarity result, the third similarity result, and the fourth similarity result so that the dimensions of the first similarity result, the second similarity result, the third similarity result, and the fourth similarity result are the same; The first similarity result, the second similarity result, the third similarity result, and the fourth similarity result are combined.

7. An electronic device, the electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 6.

8. A computer-readable medium having stored thereon computer program instructions that can be executed by a processor to implement the method as described in any one of claims 1 to 6.

9. A computer program product comprising a computer program, said computer program being processed by a processor. When executed, the method of any one of claims 1 to 6 is implemented.

Images