Image retrieval method, device, computer equipment and storage medium

Abstract

The present application discloses an image retrieval method, apparatus, computer equipment and storage medium, relating to the field of artificial intelligence. The method comprises: extracting features from a target image to obtain a first feature map of the target image, the first feature map being used to characterize the image features of a first attention area in the target image; guiding the attention of the target image based on the first feature map to obtain a guide image corresponding to the target image, the attention area of ​​the guide image being a second attention area; extracting features from the guide image to obtain a second feature map of the guide image, the second feature map being used to characterize the image features of the second attention area in the guide image; generating a first feature vector based on the first feature map, and generating a second feature vector based on the second feature map; performing image retrieval based on the first feature vector and the second feature vector to obtain an image retrieval result. The solution provided in the embodiment of the present application enriches the expression dimension of image features and improves the accuracy of image retrieval.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence, and in particular to an image retrieval method, apparatus, computer equipment, and storage medium. Background Art

[0002] Image retrieval is a technology that finds similar images from a library based on a given query image. It is often used in application scenarios such as image deduplication and image search. In these applications, similar images contain the same instances as the query image, which can be people, scenes, buildings, etc.

[0003] In related technologies, when performing image retrieval, a pre-trained feature extraction network is usually used to extract features of the image to be retrieved to obtain the features of the image to be retrieved, and then based on the features of the image to be retrieved and the image features of each image in the gallery, images with similar image features to the image to be retrieved are searched in the gallery.

[0004] However, when using the feature extraction network in related technologies for feature extraction, only the image features within a specific image area are focused on, such as only the image features in the foreground area of ​​the image, resulting in a single expression dimension of the extracted image features, which in turn affects the accuracy of image retrieval. Summary of the Invention

[0005] The embodiments of the present application provide an image retrieval method, apparatus, computer device, and storage medium that can extract multi-attention image features, enrich the expression dimensions of image features, and improve the accuracy of image retrieval. The technical solution is as follows:

[0006] In one aspect, an embodiment of the present application provides an image retrieval method, the method comprising:

[0007] Performing feature extraction on the target image to obtain a first feature map of the target image, where the first feature map is used to represent image features of a first attention area in the target image;

[0008] performing attention guidance on the target image based on the first feature map to obtain a guidance image corresponding to the target image, wherein an attention region of the guidance image is a second attention region, and the second attention region is different from the first attention region;

[0009] performing feature extraction on the guide image to obtain a second feature map of the guide image, wherein the second feature map is used to represent image features of the second attention area in the guide image;

[0010] generating a first feature vector based on the first feature map, and generating a second feature vector based on the second feature map;

[0011] An image retrieval is performed based on the first feature vector and the second feature vector to obtain an image retrieval result, wherein the image retrieval result includes similar images to the target image.

[0012] On the other hand, an embodiment of the present application provides an image retrieval device, comprising:

[0013] a feature extraction module, configured to extract features from a target image to obtain a first feature map of the target image, wherein the first feature map is used to represent image features of a first attention area in the target image;

[0014] an attention guiding module, configured to guide the attention of the target image based on the first feature map to obtain a guiding image corresponding to the target image, wherein the attention area of ​​the guiding image is a second attention area, and the second attention area is different from the first attention area;

[0015] The feature extraction module is further configured to perform feature extraction on the guide image to obtain a second feature map of the guide image, wherein the second feature map is used to represent image features of the second attention area in the guide image;

[0016] a vector generation module, configured to generate a first feature vector based on the first feature map, and generate a second feature vector based on the second feature map;

[0017] The retrieval module is configured to perform image retrieval based on the first feature vector and the second feature vector to obtain image retrieval results, wherein the image retrieval results include similar images to the target image.

[0018] On the other hand, an embodiment of the present application provides a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the image retrieval method described in the above aspect.

[0019] On the other hand, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the image retrieval method as described in the above aspects.

[0020] In another aspect, embodiments of the present application provide a computer program product or computer program, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image retrieval method provided in various optional implementations of the aforementioned aspects.

[0021] In an embodiment of the present application, when performing image retrieval on a target image, features of the target image are extracted, and attention of the target image is guided based on the extracted feature map to obtain a guide image having a different attention area from the target image, thereby performing image retrieval based on the feature vectors corresponding to the target image and the guide image to obtain an image retrieval result; the scheme provided in the embodiment of the present application is adopted to realize image feature extraction of multiple attention areas, and image retrieval is performed based on the image features corresponding to different attention areas, such as performing image retrieval based on the image features of the foreground area and the background area in the target image at the same time, thereby enriching the expression dimension of image features during image retrieval, improving the attention to different local areas in the image, and thereby improving the accuracy of image retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0023] Figure 1 A schematic diagram showing the principle of the image detection method provided in an embodiment of the present application is shown;

[0024] Figure 2 A schematic diagram showing an implementation environment provided by an exemplary embodiment of the present application is shown;

[0025] Figure 3 A flowchart of an image retrieval method provided by an exemplary embodiment of the present application is shown;

[0026] Figure 4 A schematic diagram of the structure of a feature extraction network provided by an exemplary embodiment of the present application is shown;

[0027] Figure 5 It is a schematic diagram of the structure of the residual block in the feature extraction network;

[0028] Figure 6 A flowchart of an image retrieval method provided by another exemplary embodiment of the present application is shown;

[0029] Figure 7 shows a schematic diagram of the implementation of the guided image generation process;

[0030] Figure 8 A flowchart of a feature extraction network training process provided by an exemplary embodiment of the present application is shown;

[0031] Figure 9A flowchart of a feature extraction network training process provided by another exemplary embodiment of the present application is shown;

[0032] Figure 10 is a schematic diagram of an implementation of a feature extraction network training process shown in an exemplary embodiment;

[0033] Figure 11 A flowchart of a feature extraction network training process provided by another exemplary embodiment of the present application is shown;

[0034] Figure 12 is a schematic diagram of an implementation of a feature extraction network training process shown in an exemplary embodiment;

[0035] Figure 13 A schematic structural diagram of a computer device provided by an exemplary embodiment of the present application is shown;

[0036] Figure 14 A block diagram of an image retrieval device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0037] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0038] For ease of understanding, the nouns involved in the embodiments of the present application are first explained below.

[0039] Manual labeling: refers to the process of labeling the training samples in the training data set with ground-truth by labelers before training the neural network model. The labeling labels obtained by manual labeling serve as supervision of the model output results during the model training process. Accordingly, the model training process is the process of adjusting the model parameters so that the model output results tend to the labeling labels. The manual labeling process involved in the embodiment of the present application refers to labeling whether the images in the image group are similar. For example, when the image pair contains the first image and the second image, and the labeling labels of the first image and the second image are both 1, it indicates that the first image and the second image are similar images; when the image pair contains the first image, the second image and the third image, and the labeling labels of the first image and the second image are 1, and the labeling label of the third image is 0, it indicates that the first image and the second image are similar images, and the third image is not similar to the first image and the second image.

[0040] Embedding: In deep learning, the process of mapping high-dimensional raw data into a low-dimensional space to obtain a low-dimensional feature vector is called embedding. In some embodiments, the embedding process can be implemented using a fully connected (FC) layer. The embedding process in the embodiments of the present application refers to the process of converting the high-dimensional feature map extracted by the feature extraction network into a low-dimensional feature vector.

[0041] A mask is a binary image composed of 0s and 1s, where 1-valued areas are processed and 0-valued areas are blocked. The mask involved in the embodiments of the present application is used to process an image, thereby blocking the image corresponding to the 0-valued areas of the mask.

[0042] Guiding image: An image obtained by guiding attention to an original image, wherein the original image and the guiding image focus on different attention areas. In some embodiments, the attention areas of the original image and the guiding image complement each other. For example, when the attention area of ​​the original image is the foreground, the attention area of ​​the guiding image obtained through attention guidance is the background; for another example, when the attention area of ​​the original image is the left image area, the attention area of ​​the guiding image obtained through attention guidance is the right image area.

[0043] Loss function: Also known as the cost function, it is a function used to evaluate the degree of difference between the predicted value and the true value of a neural network model. The smaller the loss function, the better the performance of the neural network model. The model training process is the process of minimizing the loss function by adjusting the model parameters. Different neural network models use different loss functions. Common loss functions include 0-1 loss function, absolute value loss function, logarithmic loss function, exponential loss function, perceptual loss function, cross entropy loss function, KL divergence (Kullback-Leibler divergence) loss function, triplet loss function, and so on.

[0044] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0045] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0046] Computer vision (CV) is the science of making machines "see." Specifically, it refers to using cameras and computers to replace the human eye in identifying, tracking, and measuring objects. It also performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems that can extract information from images or multidimensional data. Computer vision technologies generally include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and other technologies. It also includes common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0047] The image retrieval method provided in the embodiment of the present application introduces an attention guidance mechanism. During the image retrieval process, a guidance image having different attention areas from the target image to be retrieved is generated, and feature vectors corresponding to the target image and the guidance image are obtained respectively, thereby realizing image feature extraction of multiple attention areas, and then performing image retrieval based on the feature vectors of the two, thereby increasing the attention paid to each local area during the image retrieval process, thereby improving the accuracy of image retrieval. Figure 1 A schematic diagram showing the principle of the image detection method provided in an embodiment of the present application is shown.

[0048] like Figure 1 As shown, when performing image retrieval on the target image 11, the computer device first extracts features of the target image 11 and obtains features representing the first attention area (such as Figure 1 The first feature of the image feature of the person in the foreground Figure 12 .

[0049] Get the first feature Figure 12 After that, on the one hand, the computer device has the first characteristic Figure 12 An embedding process is performed to obtain a first eigenvector 13 corresponding to the target image 11 .

[0050] On the other hand, the computer device utilizes the first feature Figure 12 The target image 11 is guided by features to obtain a guide image 14 (the guide image 14 does not include the person in the first attention area corresponding to the target image 11), and the guide image 14 is extracted to obtain a characterization of the second attention area (such as Figure 1 The second feature map 15 is obtained by performing embedding processing on the second feature map 15 to obtain a second feature vector 16 corresponding to the guide image 14.

[0051] Furthermore, the computer device performs an image search in the image library 17 based on the first feature vector 13 and the second feature vector 16 to determine similar images 18 to the target image 11 .

[0052] Through the above scheme, the image retrieval process not only focuses on the foreground area, but also the background area, thereby increasing the attention paid to each area in the image. Accordingly, when performing image retrieval, image retrieval can be performed based on image features of multiple areas, which helps to improve the accuracy of image retrieval.

[0053] The image retrieval method provided in the embodiments of the present application can be applied to scenarios with image retrieval requirements, such as image search scenarios, image deduplication scenarios, and image management scenarios, etc. The image retrieval process in different application scenarios is described below.

[0054] 1. Image search scenario

[0055] In a possible implementation, the image retrieval method provided in the embodiments of the present application may be implemented as an image retrieval system or a part of the system, and deployed in a server of an application or website that provides image retrieval services.

[0056] Upon receiving an image search request from a terminal, the server obtains the image to be searched contained in the request, generates a guide image corresponding to the image to be searched using the solution provided in the embodiments of the present application, and performs feature extraction on the image to be searched and the guide image to obtain two sets of feature vectors. The server's image library stores source images and their corresponding feature vectors. Based on the two sets of feature vectors obtained, the server searches the image library for source images whose vector distance to the image to be searched is less than a threshold, and feeds this back to the terminal for display.

[0057] 2. Image deduplication scenarios

[0058] In one scenario, the server can employ the solution provided in the embodiments of this application to generate a feature vector (composed of two sets of feature vectors) corresponding to each material image in the image library. Based on this feature vector, the server can check for duplicates of existing material images in the image library, for example, identifying material images with too small a vector distance as duplicates. For multiple duplicate material images, the server can retain only one, thereby freeing up storage space.

[0059] In another possible scenario, when a new image needs to be added to the gallery, the server uses the solution provided in the embodiment of the present application to determine the feature vector of the new image, and based on the feature vector, determines whether there are duplicate images in the gallery. If so, the new image is refused to be added to the gallery. If not, the new image and its corresponding feature vector are associated and stored in the gallery.

[0060] 3. Image management scenario

[0061] In a possible implementation, the image retrieval method provided in the embodiment of the present application can be implemented as a part of a picture management application or program and installed in a terminal.

[0062] For each stored picture, the terminal can generate a feature vector corresponding to each picture through the picture management application, and based on the feature vector of each picture, identify similar pictures with similar instances, thereby grouping similar pictures into the same album. Optionally, the terminal can further generate an album name for the album based on the instances contained in each picture in the same album, to facilitate subsequent user search.

[0063] It should be noted that the above application scenarios are only for illustrative purposes and do not limit the actual application scenarios of the embodiments of the present application.

[0064] Figure 2 A schematic diagram of an implementation environment provided by an exemplary embodiment of the present application is shown. This embodiment is described using an image retrieval method applied to an image retrieval system as an example. The implementation environment includes a terminal 210 and a server 220. Data communication between the terminal 210 and the server 220 is performed via a communication network. Optionally, the communication network can be either a wired network or a wireless network, and the communication network can be at least one of a local area network, a metropolitan area network, and a wide area network.

[0065] The terminal 210 is an electronic device for image retrieval, which may be a mobile terminal such as a smart phone, a tablet computer, a laptop computer, or a desktop computer, a projection computer, a smart TV, etc., which is not limited in the present embodiment.

[0066] In a possible implementation, the terminal 210 may implement image retrieval through an installed image retrieval application, or implement image retrieval by accessing an image retrieval website.

[0067] Server 220 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. In the embodiment of the present application, server 220 is a server for providing image retrieval services.

[0068] In one possible implementation, server 220 is provided with an image library and a multi-attention feature extraction network for extracting features from images. The image library stores source images and corresponding multi-attention feature vectors for each source image. These multi-attention feature vectors are obtained by pre-extracting features from the source images using the multi-attention feature extraction network. In some embodiments, the source images and their corresponding multi-attention features can be stored in a blockchain.

[0069] In a possible application scenario, such as Figure 2 As shown, the terminal 210 accesses the image retrieval website and sends an image retrieval request containing the image to be retrieved to the website's server 220. The server 220 first performs multi-attention feature extraction on the image to be retrieved through the multi-attention feature extraction network 221 to obtain a multi-attention feature vector. Then, based on the multi-attention feature vector, similar images are searched from the gallery 222, and the similar images are fed back to the terminal 210 as retrieval results.

[0070] In other possible implementations, the multi-attention feature extraction network can also be deployed in the terminal 210. Accordingly, the terminal 210 can perform feature extraction on the image to be retrieved locally to obtain a multi-attention feature vector, and send an image retrieval request containing the multi-attention feature vector to the server 220, thereby reducing the computing pressure of the server 220. This embodiment does not limit this.

[0071] In addition, the above-mentioned multi-attention feature extraction network can be trained by the server 220, or it can be deployed on the server 220 after being trained by other devices. For the convenience of description, the following embodiments are applied to a computer device (which can be Figure 2 The example is explained below, where the server or terminal in the system is a server, and the model training is performed by a computer device.

[0072] Figure 3A flowchart of an image retrieval method provided by an exemplary embodiment of the present application is shown. This embodiment uses the method applied to a computer device as an example to illustrate the method, which includes the following steps.

[0073] Step 301 : extract features from a target image to obtain a first feature map of the target image. The first feature map is used to represent image features of a first attention area in the target image.

[0074] Optionally, the target image is a local image, such as an image in a local gallery, or an image obtained from another terminal, such as an image included in an image retrieval request sent by the terminal. This embodiment of the application limits the source of the target image.

[0075] Regarding the feature extraction method, in one possible implementation, a computer device performs feature extraction on a target image through a pre-trained feature extraction network to obtain a first feature map. The size of the first feature map is w*h*c, where w is the width of each channel feature map, h is the height of each channel feature map, and c is the number of channels of the feature map.

[0076] Similar to the focus of human eyes when viewing an image, a computer device has a key feature extraction area when extracting features from a target image. This key feature extraction area is the first attention area. Therefore, the first feature map has a better ability to express image features for the first attention area, but has a poorer ability to express image features for areas outside the first attention area in the target image.

[0077] In some embodiments, the first attention area is the foreground area of ​​the target image, or the first attention area is the area corresponding to a specific entity (such as a person, animal, etc.) in the target image; and the first attention area can be a continuous area or a discontinuous area, which is not limited in this embodiment.

[0078] like Figure 1 As shown, the first attention area in the target image 11 is the area where the person in the foreground is located.

[0079] In step 302 , attention is guided on the target image based on the first feature map to obtain a guidance image corresponding to the target image. The attention area of ​​the guidance image is a second attention area, and the second attention area is different from the first attention area.

[0080] In order to realize multi-attention image feature extraction, in an embodiment of the present application, the computer device determines the first attention area to be focused on during feature extraction based on the first feature map, and thus guides attention based on the first attention area to generate a guide image corresponding to different attention areas of the target image.

[0081] The size of the guide image is the same as that of the target image. Optionally, the guide image is an image obtained by masking the first attention area in the target image, that is, the first attention area corresponding to the target image and the second attention area corresponding to the guide image are complementary.

[0082] In some embodiments, when the first attention area is the foreground area of ​​the target image, the second attention area is the background area of ​​the target image.

[0083] Indicative, such as Figure 1 As shown, the second attention area in the guiding image 14 is the area where the pond is located in the background, and the area where the person is located in the foreground is shielded.

[0084] Step 303 : extract features from the guide image to obtain a second feature map of the guide image. The second feature map is used to represent image features of the second attention area in the guide image.

[0085] Since the attention area of ​​the guide image is the second attention area, when the computer extracts features from the guide image, the key feature extraction area of ​​the computer device when extracting features from the guide image is located in the second attention area. Accordingly, the second feature map has better image feature expression capabilities for the second attention area.

[0086] Regarding the feature extraction method, in one possible implementation, the computer device uses the same feature extraction network used to extract the first feature map to extract features from the guidance image, thereby obtaining a second feature map. The second feature map has the same size and number of channels as the first feature map.

[0087] Indicative, such as Figure 1 As shown, the first feature Figure 12 It has the same size and number of channels as the second feature map 15.

[0088] Step 304: Generate a first feature vector based on the first feature map, and generate a second feature vector based on the second feature map.

[0089] Since the first feature map and the second feature map obtained by feature extraction are high-dimensional data, they cannot be directly used for image retrieval. Therefore, the computer device needs to perform data dimensionality reduction to convert the high-dimensional feature map data into a low-dimensional feature vector.

[0090] In a possible implementation, the computer device performs embedding processing on the first feature map and the second feature map respectively to obtain a first feature vector corresponding to the target image and a second feature vector corresponding to the guide image.

[0091] The first eigenvector is used to characterize the image features of the first attention area, and the second eigenvector is used to characterize the image features of the second attention area.

[0092] Regarding the specific method of embedding processing, in one possible implementation, the pre-trained feature extraction network includes an FC layer, and the computer device performs embedding processing on the feature map through the FC layer to obtain a feature vector.

[0093] Step 305 : Perform image retrieval based on the first eigenvector and the second eigenvector to obtain image retrieval results, where the image retrieval results include images similar to the target image.

[0094] After obtaining feature vectors representing different attention areas through the above steps, the computer device performs image retrieval based on these feature vectors to identify similar images to the target image. Because the image features of different attention areas are utilized during the retrieval, the retrieved similar images not only have similar image features to the first attention area in the target image, but also have similar image features to the second attention area.

[0095] In one possible implementation, the computer device fuses the first feature vector and the second feature vector, thereby performing image retrieval based on the fused feature vector. During the image retrieval process, the computer device uses the vector distance between the feature vectors of the images as a basis to retrieve images with similar feature vectors.

[0096] Compared to image retrieval based on single-attention image features in related technologies, the embodiments of the present application utilize multiple-attention image features, making it possible to perform retrieval based on image features from more local regions, thereby improving the accuracy of image retrieval. For example, the solution provided by the embodiments of the present application can retrieve images with similar foregrounds and similar backgrounds to the target image, while the solution provided by related technologies can only retrieve images with similar foregrounds, including a large number of images with very different backgrounds, resulting in a large number of image retrieval results but a low accuracy rate.

[0097] To sum up, in the embodiments of the present application, when performing image retrieval on a target image, features of the target image are extracted, and attention of the target image is guided based on the extracted feature map to obtain a guide image having a different attention area from the target image, thereby performing image retrieval based on the feature vectors corresponding to the target image and the guide image to obtain an image retrieval result; the scheme provided in the embodiments of the present application is adopted to realize image feature extraction of multiple attention areas, and image retrieval is performed based on the image features corresponding to different attention areas, such as performing image retrieval based on the image features of the foreground area and the background area in the target image at the same time, thereby enriching the expression dimension of image features during image retrieval, improving the attention to different local areas in the image, and thereby improving the accuracy of image retrieval.

[0098] In one possible embodiment, a pre-trained feature extraction network is provided in the computer device, and the feature extraction network is composed of several convolutional layers and fully connected layers, wherein feature extraction of the target image and the guide image is performed by the convolutional layer (that is, the first feature map and the second feature map are extracted by the convolutional layer), and vectorization processing of the feature map is performed by the fully connected layer (that is, the first feature vector and the second feature vector are generated by the fully connected layer based on the first feature map and the second feature map).

[0099] In some embodiments, the convolutional layer of the feature extraction network uses the parameters of ResNet101 trained on the ImageNet dataset, and an FC layer is added after the last convolutional layer. Schematically, the structure of the feature extraction network is as follows: Figure 4 shown.

[0100] The feature extraction network includes five convolutional layers and one fully connected layer, and the convolutional layer 3 contains 3 residual blocks, the convolutional layer 3 contains 4 residual blocks, the convolutional layer 4 contains 23 residual blocks, the convolutional layer 5 contains 3 residual blocks (the convolutional layer 5 is used to output the first feature map and the second feature map), and the fully connected layer outputs a 1×128 feature vector. In an illustrative example, the structure of each residual block in the convolutional layer 2 is as follows: Figure 5 As shown in the figure, the input data of the residual block is 256-dimensional. The first residual layer contains 64 1×1 convolution kernels, the second residual layer contains 64 3×3 convolution kernels, and the third residual layer contains 256 1×1 convolution kernels. The data between residual layers is activated by Relu (Reliant Unit) before being input into the next residual layer. The output of the residual block is the concatenation of the input data and the output data of the third residual layer, which is also activated by Relu. During network training, a Gaussian distribution with a variance of 0.01 and a mean of 0 is used for initialization.

[0101] Figure 6A flowchart of an image retrieval method provided by another exemplary embodiment of the present application is shown. This embodiment uses the method applied to a computer device as an example to illustrate the method, which includes the following steps.

[0102] Step 601: extract features of a target image through a convolutional layer of a feature extraction network to obtain a first feature map of the target image. The first feature map is used to represent image features of a first attention area in the target image.

[0103] Optionally, the computer device inputs the target image into a feature extraction network, and each convolution layer in the feature extraction network performs convolution processing in sequence, and uses the feature map output by the last convolution layer as the first feature map.

[0104] Indicatively, when using Figure 4 When the feature extraction network is shown, the computer device determines the 19*32*2048 feature map (2048 is the number of channels) output by the convolutional layer 5 as the first feature map.

[0105] Step 602 : Generate a first mask corresponding to the target image based on the first feature map, where the attention area represented by the first mask is the first attention area.

[0106] In this embodiment, the computer device generates the guiding image corresponding to the target image by using a mask shielding method, so the computer device first needs to determine a first mask for representing the first attention area.

[0107] Optionally, the first mask has the same size as the single-channel feature map in the first feature map. Furthermore, to facilitate subsequent masking, the first mask is a binary mask, that is, the values ​​in the first mask are 0 and 1.

[0108] In a possible implementation, this step may include the following steps.

[0109] 1. Generate an original mask based on the first feature map of each channel, where the values ​​in the original mask are continuous values.

[0110] Optionally, for each spatial position in the first feature map, the computer device calculates the average value of each channel, and uses the average value as the value corresponding to the spatial position in the original mask.

[0111] With reference to the example in step 601 , when the first feature map is 19*32*2048, the computer device calculates the average value of 2048 channels to generate a 19*32 original mask.

[0112] 2. Based on the numerical average value of the original mask, each value in the original mask is binarized to obtain a first mask.

[0113] Furthermore, the computer device calculates the numerical average of each value in the original mask and compares the size relationship between each value and the numerical average. If the value is greater than the numerical average, it indicates that the image position corresponding to the spatial position is in the attention area (the extracted image features are reconstructed), and the value is set to 1. If the value is less than the numerical average, it indicates that the image position corresponding to the spatial position is outside the attention area (inadequate image features are extracted), and the value is set to 0.

[0114] In combination with the example in the above steps, when the original mask is 19*32, the computer device calculates the numerical average of 608 values, and binarizes the values ​​in the original mask based on the numerical average to obtain a 19*32 first mask consisting of 0 and 1.

[0115] Step 603: Generate a second mask based on the first mask, and the attention area represented by the second mask is the second attention area.

[0116] Since the area corresponding to the value 1 in the first mask represents the first attention area, and the area corresponding to the value 0 represents the second attention area, in a possible embodiment, the computer device can obtain a second mask by inverting the values ​​in the first mask, wherein the area corresponding to the value 1 in the second mask represents the second attention area, and the area corresponding to the value 0 represents the first attention area.

[0117] Step 604: Generate a guidance image based on the second mask and the target image.

[0118] Furthermore, the computer device performs masking processing on the first attention area in the target image based on the second mask, thereby generating a guide image containing only the second attention area.

[0119] Because the second mask belongs to the feature space and the target image belongs to the image space, before performing mask processing, the computer device must first reverse-map the values ​​in the second mask from the feature space to the image space. This means determining the corresponding positions in the target image to the spatial locations in the second mask, and obtaining a guidance image with the same size as the target image. The target image can then be masked using the image mask to generate the guidance image. This reverse mapping process is the inverse of the image feature extraction process.

[0120] Optionally, when masking the target image using the image mask, the computer device shields corresponding pixel points in the target image based on the positions of pixel points with pixel values ​​of 0 in the image mask.

[0121] Schematically, the process of generating a guidance image based on a target image is as follows: Figure 7As shown, the computer device first generates a first image mask 72 corresponding to the attention area of ​​the target image 71. The area where the dog is located in the first image mask 72 is the non-masked area. Then, the computer device performs an inversion process on the first image mask 72 to obtain a second image mask 73. The area where the dog is located in the second image mask 73 is the masked area. Finally, the computer device uses the second image mask 73 to mask the target image 71, that is, to mask the area where the dog is located in the target image 71, thereby obtaining a guidance image 74.

[0122] Step 605 , extracting features from the guidance image through the convolutional layer of the feature extraction network to obtain a second feature map of the guidance image, where the second feature map is used to represent image features of the second attention area in the guidance image.

[0123] Optionally, the computer device guides the image into the feature extraction network, and the convolution layers in the feature extraction network perform convolution processing in sequence, and the feature map output by the last convolution layer is used as the second feature map. Figure 4 When the feature extraction network is shown, the computer device determines the 19*32*2048 feature map (2048 is the number of channels) output by the convolution layer 5 as the second feature map.

[0124] Step 606 : Generate a first feature vector based on the first feature map and generate a second feature vector based on the second feature map through a fully connected layer of the feature extraction network.

[0125] Furthermore, the computer device inputs the first feature map and the second feature map into a fully connected layer of a feature extraction network, and the fully connected layer performs embedding processing on the first feature map and the second feature map to obtain a first feature vector and a second feature vector.

[0126] Indicatively, when using Figure 4 When the feature extraction network is shown, the first feature vector and the second feature vector output by the computer device are both 1×128-dimensional feature vectors.

[0127] Step 607: concatenate the first eigenvector and the second eigenvector to obtain a target fused eigenvector.

[0128] In this embodiment, the computer device performs vector fusion on the first feature vector and the second feature vector by concatenation to obtain a target fused feature vector.

[0129] Illustratively, when the first eigenvector and the second eigenvector are both 1×128-dimensional eigenvectors, the target eigenvector obtained by concatenation is a 1×256-dimensional eigenvector.

[0130] Step 608: Determine the vector distance between the target fused feature vector and the corresponding fused feature vector of the image in the image library.

[0131] In one possible implementation, a library stores correspondences between each source image and a fused feature vector. This fused feature vector is obtained by pre-extracting features from the source images using a feature extraction network. Accordingly, when searching the library for images similar to a target image, the computer calculates the vector distance between the target fused feature vector and the corresponding fused feature vectors of each source image, thereby identifying similar images based on this vector distance. The vector distance is negatively correlated with the similarity between images; that is, smaller vector distances indicate higher image similarity, while larger vector distances indicate lower image similarity.

[0132] Optionally, the vector distance may be cosine distance, Mahalanobis distance, KL distance, or L2 norm distance, etc. The embodiment of the present application does not limit the specific type of vector distance.

[0133] Step 609: Determine the image retrieval result based on the vector distance.

[0134] In a possible implementation, the computer device determines an image whose vector distance is less than a preset vector distance as a similar image to the target image. For example, the preset vector distance is 0.1.

[0135] In other possible implementations, the computer device may further sort the images in the image library based on ascending order of vector distance, thereby selecting the top k% images as similar images. This embodiment does not limit the specific method of determining image retrieval results based on vector distance.

[0136] In one possible implementation, when a target image needs to be added to a gallery, the computer device, after obtaining image retrieval results based on the feature vector, determines whether there are any duplicate images in the gallery. Images whose vector distance to the target image is less than a distance threshold (e.g., 0.01) are determined to be duplicate images. If there are no duplicate images in the gallery, the computer device associates the target image and the target fusion feature vector and stores them in the gallery for subsequent image retrieval. If there are duplicate images in the gallery, the computer device does not store the target image in the gallery.

[0137] In this embodiment, the computer device generates a first mask representing the first attention area based on the feature map of the target image, and uses the first mask to further generate a second mask representing the second attention area, and then uses the second mask to extract features of the target image to obtain a guiding image that masks the first attention area. This eliminates the need for the user to manually specify the attention area in the target image, thereby improving the generation quality and efficiency of the guiding image.

[0138] The above embodiments illustrate the image retrieval process. Since a feature extraction network is required to extract image features during image retrieval, the computer must pre-train the feature extraction network to ensure the quality of feature extraction and, therefore, subsequent retrieval. The following describes the feature extraction network training process using an exemplary embodiment.

[0139] Figure 8 A flowchart of a feature extraction network training process provided by an exemplary embodiment of the present application is shown. This embodiment uses the method applied to a computer device as an example to illustrate the method, which includes the following steps.

[0140] Step 801: extract features from the sample image group through a feature extraction network to obtain a sample feature map group corresponding to the sample image group. The sample image group includes at least two sample images, and the sample feature map group includes a sample feature map corresponding to each sample image.

[0141] In one possible embodiment, during the training process, the computer device inputs each sample image in the sample image group into the feature extraction network respectively, and the convolution layer in the feature extraction network performs feature extraction, thereby obtaining a sample feature map corresponding to each sample image, and then obtaining a sample feature map group.

[0142] The sample image group includes pre-labeled images, and the same sample image group contains at least two sample images, and at least two sample images contain two similar sample images (similar sample images have the same label). It should be noted that the sample images are manually labeled, and the manual labeling focuses on the similarity of each region in the sample images.

[0143] Optionally, a sample image group includes two similar sample images, or a sample image group includes two similar sample images and a sample image that is not similar to the other two sample images (has a different label from the other two sample images).

[0144] Step 802 : Based on the sample feature map group, attention is guided on the sample image group to obtain a sample guidance image group corresponding to the sample image group. The sample guidance image group includes a sample guidance image corresponding to each sample image.

[0145] Similar to the image retrieval process, during the training process, the computer equipment needs to generate a corresponding sample guidance image for each sample image, so as to subsequently extract image features of different attention areas based on the sample guidance image.

[0146] In one possible embodiment, the computer device determines the first sample attention area in each sample image based on each sample feature map in the sample feature map group, and thus guides attention based on the first sample attention area, generates a sample guidance image (corresponding to the second sample attention area) corresponding to a different attention area of ​​the sample image, and then obtains a sample guidance image group corresponding to the sample image group.

[0147] Step 803 , extracting features from the sample guidance image group through a feature extraction network to obtain a guidance feature map group corresponding to the sample guidance image group, wherein the guidance feature map group includes a sample guidance feature map corresponding to each sample guidance image.

[0148] In one possible implementation, a computer device inputs each sample guidance image in the sample guidance image group into a feature extraction network, and the convolutional layer in the feature extraction network performs feature extraction, thereby obtaining a sample guidance feature map corresponding to each sample guidance image, and then obtaining a guidance feature map group.

[0149] Step 804: Generate a sample feature vector group corresponding to the sample feature map group and a guide feature vector group corresponding to the guide feature map group through a feature extraction network. The sample feature vector group includes the sample feature vectors of each sample image, and the guide feature vector group includes the guide feature vectors of each sample guide image.

[0150] In one possible implementation, the computer device performs dimensionality reduction processing (embedding processing) on ​​each sample feature map in the sample feature map group through the fully connected layer in the feature extraction network to obtain the sample feature vector corresponding to each sample image, and then generates a sample feature vector group; and performs dimensionality reduction processing (embedding processing) on ​​each sample guide feature map in the guide feature map group through the fully connected layer in the feature extraction network to obtain the guide feature vector corresponding to each sample guide image, and then generates a guide feature vector group.

[0151] In some embodiments, when the sample image group contains two sample images, the obtained sample feature vector group contains two sample feature vectors, and the guided feature vector group contains two guided feature vectors; when the sample image group contains three sample images, the obtained sample feature vector group contains three sample feature vectors, and the guided feature vector group contains three guided feature vectors.

[0152] Step 805: training a feature extraction network based on the sample feature vector group and the guide feature vector group.

[0153] In one possible implementation, the computer device uses the labels corresponding to the sample images in the sample image group as supervision, determines the loss of the sample image group based on the sample feature vector group and the guide feature vector group, and thus trains the feature extraction network with the goal of minimizing the loss.

[0154] Optionally, the loss is determined based on the vector distance between feature vectors in the sample feature vector group and the guide feature vector group.

[0155] Optionally, the computer device uses a gradient descent method based on stochastic gradient descent (SGD) to solve the weights and bias parameters of the feature extraction network. The embodiment of the present application does not limit the specific training method of the feature extraction network.

[0156] When the feature extraction network meets the training completion conditions (such as loss convergence or reaching the number of iterative training times), the computer device stops training and uses the test set to test the trained feature extraction network, and fine-tunes the feature extraction network based on the test results.

[0157] In this embodiment, a two-stage attention mechanism is used in training to mine more local image features in the sample image, thereby realizing the secondary utilization of the sample image. On the one hand, it enables the feature extraction network to learn the image features of multiple main attention areas during the training process, and on the other hand, it can improve the utilization rate of the labeled data corresponding to the sample image.

[0158] For sample image groups of different forms, the loss functions constructed by the computer device when performing network training are also different. The following uses exemplary embodiments to illustrate the network training process when using sample image groups of different forms.

[0159] Figure 9 A flowchart of a feature extraction network training process provided by an exemplary embodiment of the present application is shown. This embodiment uses the method applied to a computer device as an example to illustrate the method, which includes the following steps.

[0160] Step 901 : extract features from a sample image group through a feature extraction network to obtain a sample feature map group corresponding to the sample image group. The sample image group includes a first sample image and a second sample image. The first sample image is similar to the second sample image.

[0161] In this embodiment, the first sample image and the second sample image have the same label. In a possible implementation, the second sample image is obtained by cropping or slightly adjusting the first sample image.

[0162] Indicative, such as Figure 10 As shown, the sample image group includes a first sample image 1001 and a second sample image 1002, wherein the second sample image 1002 is cropped based on the first sample image 1001. Feature extraction is performed through a feature extraction network 1003 to obtain a first sample feature map 1004 corresponding to the first sample image 1001 and a second sample feature map 1005 corresponding to the second sample image 1002.

[0163] Step 902: Based on the sample feature map group, attention is guided on the sample image group to obtain a sample guided image group corresponding to the sample image group, including a first sample guided image and a second sample guided image.

[0164] The computer device guides attention based on the first sample feature map to obtain a first sample guidance map, and the computer device guides attention based on the second sample feature map to obtain a second sample guidance map.

[0165] Indicative, such as Figure 10 As shown, the computer device uses the first sample feature map 1004 to guide attention and obtains the first sample guidance image 1006, and uses the second sample feature map 1005 to guide attention and obtains the second sample guidance image 1007.

[0166] Step 903 : extract features from the sample guidance image group through a feature extraction network to obtain a guidance feature map group corresponding to the sample guidance image group, wherein the guidance feature map group includes a first sample guidance feature map and a second sample guidance feature map.

[0167] The computer device extracts features from the first sample guide image and the second sample guide image respectively through a feature extraction network to obtain a first sample guide feature map and a second sample guide feature map.

[0168] Indicative, such as Figure 10 As shown, the computer device performs feature extraction on the first sample guide image 1006 and the second sample guide image 1007 through the feature extraction network 1003 to obtain a first sample guide feature map 1008 and a second sample guide feature map 1009.

[0169] Step 904: Generate a sample feature vector group corresponding to the sample feature map group and a guided feature vector group corresponding to the guided feature map group through a feature extraction network. The sample feature vector group includes a first sample feature vector and a second sample feature vector, and the guided feature vector group includes a first guided feature vector and a second guided feature vector.

[0170] Furthermore, the computer device performs embedding processing on the first sample feature map, the second sample feature map, the first sample guided feature map, and the second sample guided feature map through a feature extraction network to obtain a first sample feature vector corresponding to the first sample image, a second sample feature vector corresponding to the second sample image, a first guided feature vector corresponding to the first sample guided image, and a second guided feature vector corresponding to the second sample guided image.

[0171] Indicative, such as Figure 10 As shown, the computer device performs embedding processing on the extracted feature maps to obtain a first sample feature vector 1010, a second sample feature vector 1011, a first guide feature vector 1012 and a second guide feature vector 1013.

[0172] Step 905 : Determine the sample feature loss based on the first sample feature vector and the second sample feature vector in the sample feature vector group.

[0173] In one possible implementation, the computer device calculates a KL divergence between the first sample feature vector and the second sample feature vector, thereby determining the KL divergence as the sample feature loss. Because the first sample image and the second sample image are similar, the sample feature loss will continue to decrease during network training, that is, the KL divergence between the first sample feature vector and the second sample feature vector will continue to decrease.

[0174] Optionally, the KL divergence between eigenvectors is calculated as follows:

[0175]

[0176] Among them, p(x i ) and q(x i ) are the sample feature vectors corresponding to the two sample images in the i-th sample image group, and N is the number of sample image groups.

[0177] Indicative, such as Figure 10 As shown, the computer device determines the KL divergence between the first sample feature vector 1010 and the second sample feature vector 1012 as the sample feature loss 1014 .

[0178] Step 906 : Determine a guided feature loss based on the first guided feature vector and the second guided feature vector in the guided feature vector group.

[0179] The aforementioned sample feature loss is the feature loss of the first sample attention region between different sample images. In this embodiment, since a feature vector representing the features of the second sample attention region is obtained, the computer device can calculate the KL divergence between the first guided feature vector and the second guided feature vector, thereby determining this KL divergence as the guided feature loss. Since the first and second sample images are similar, the features of their second sample attention regions should also be similar. Therefore, the guided feature loss will continue to decrease during network training, that is, the KL divergence between the first guided feature vector and the second guided feature vector will continue to decrease.

[0180] Indicative, such as Figure 10 As shown, the computer device determines the KL divergence between the first guided feature vector 1012 and the second guided feature vector 1013 as the guided feature loss 1015.

[0181] Step 907: Determine the fusion feature loss based on the sample feature vector group and the guide feature vector group.

[0182] Steps 905 and 906 above focus on the first sample attention area and the second sample attention area, respectively. In this step, the computer device fuses the feature vectors in the sample feature vector group and the guide feature vector group, and determines the fusion feature loss based on the fused feature vectors, thereby achieving the effect of simultaneously focusing on the first sample attention area and the second sample attention area. Optionally, this step may include the following steps.

[0183] 1. Concatenate the first sample feature vector and the first guide feature vector to obtain a first fused feature vector.

[0184] The computer device splices the feature vectors corresponding to the first sample image and the first sample guide image to obtain a first fused feature vector, which can simultaneously represent image features of different attention areas in the first sample image.

[0185] Indicative, such as Figure 10 As shown, the computer device concatenates the first sample feature vector 1010 and the first guide feature vector 1012 to obtain a first fused feature vector 1016 .

[0186] 2. Concatenate the second sample feature vector and the second guide feature vector to obtain a second fused feature vector.

[0187] The computer device splices the feature vectors corresponding to the second sample image and the second sample guide image to obtain a second fused feature vector, which can simultaneously represent image features of different attention areas in the second sample image.

[0188] Indicative, such as Figure 10 As shown, the computer device concatenates the second sample feature vector 1011 and the second guide feature vector 1013 to obtain a second fused feature vector 1017 .

[0189] 3. Determine the KL divergence loss of the first fused feature vector and the second fused feature vector as the fused feature loss.

[0190] The computer device may calculate a KL divergence between the first fused feature vector and the second fused feature vector, thereby determining the KL divergence as a fused feature loss. Since the first sample image and the second sample image are similar images, the features of the attention regions corresponding to the first and second sample images should be similar. Therefore, the fused feature loss will continue to decrease during network training, that is, the KL divergence between the first fused feature vector and the second fused feature vector will continue to decrease.

[0191] Indicative, such as Figure 10 As shown, the computer device determines the KL divergence between the first fused feature vector 1016 and the second fused feature vector 1017 as the fused feature loss 1018 .

[0192] In step 908, a feature extraction network is trained based on the sample feature loss, the guided feature loss, and the fusion feature loss. The sample feature loss, the guided feature loss, and the fusion feature loss are all KL divergence losses.

[0193] In this embodiment, the computer device calculates the total loss of the sample feature loss, the guided feature loss, and the fusion feature loss, thereby training the feature extraction network based on the total loss.

[0194] Schematically, the total loss of the feature extraction network can be expressed as: Loss = loss_ori + loss_attention + loss_concat, where loss_ori is the sample feature loss, loss_attention is the guided feature loss, and loss_concat is the fusion feature loss.

[0195] In this embodiment, the computer device determines the sample feature loss, the guided feature loss, and the fusion feature loss based on the sample feature vector group and the guided feature vector group, and then uses the total loss of the three to train the feature extraction network, so that the feature extraction network can learn the image features of different local areas in the sample image, which helps to improve the accuracy and comprehensiveness of feature extraction when the feature extraction network is subsequently used for feature extraction.

[0196] Figure 11 A flowchart of a feature extraction network training process provided by another exemplary embodiment of the present application is shown. This embodiment uses the method applied to a computer device as an example to illustrate the method, which includes the following steps.

[0197] Step 1101: extract features from the sample image group through a feature extraction network to obtain a sample feature map group corresponding to the sample image group. The sample image group includes a first sample image and a second sample image. The first sample image is similar to the second sample image, and the third sample image is dissimilar to both the first sample image and the second sample image.

[0198] In this embodiment, the first and second sample images have the same label, while the third sample image has a different label from the first and second sample images. That is, the sample image group consists of an anchor, a positive sample, and a negative sample. For example, the first and second sample images have a label of 1, while the third sample image has a label of 0.

[0199] Regarding the selection of negative samples in the sample image group, in one possible implementation, when each round of training includes bs sample image pairs (each sample image pair contains two similar images), the computer device randomly selects a sample image from the bs sample image pairs to obtain bs sample images, thereby calculating the embedding similarity between the bs sample images, and then determining the negative samples based on the embedding similarity (for example, the similarity ranking is in the bottom 10%). Furthermore, the computer device combines the sample image pairs and the corresponding negative samples to obtain a ternary sample image group.

[0200] Indicative, such as Figure 12 As shown, the sample image group includes a first sample image 1201, a second sample image 1202, and a third sample image 1203. The second sample image 1202 is cropped based on the first sample image 1201, and the two are similar images. The third sample image 1203 is dissimilar to both the first sample image 1201 and the second sample image 1202. Feature extraction is performed through a feature extraction network 1204 to obtain a first sample feature map 1205 corresponding to the first sample image 1201, a second sample feature map 1206 corresponding to the second sample image 1202, and a third sample feature map 1207 corresponding to the third sample image 1203.

[0201] Step 1102 : Based on the sample feature map group, attention is guided on the sample image group to obtain a sample guided image group corresponding to the sample image group, including a first sample guided image, a second sample guided image, and a third sample guided image.

[0202] The computer device guides attention based on the first sample feature map to obtain a first sample guidance map, guides attention based on the second sample feature map to obtain a second sample guidance map, and guides attention based on the third sample feature map to obtain a third sample guidance map.

[0203] Indicative, such as Figure 12 As shown, the computer device uses the first sample feature map 1205 to guide attention and obtains the first sample guidance image 1208, uses the second sample feature map 1206 to guide attention and obtains the second sample guidance image 1209, and uses the third sample feature map 1207 to guide attention and obtains the third sample guidance image 1210.

[0204] Step 1103 , extracting features from the sample guidance image group through a feature extraction network to obtain a guidance feature map group corresponding to the sample guidance image group, wherein the guidance feature map group includes a first sample guidance feature map, a second sample guidance feature map, and a third sample guidance feature map.

[0205] The computer device extracts features from the first sample guide image, the second sample guide image, and the third sample guide image through a feature extraction network to obtain a first sample guide feature map, a second sample guide feature map, and a third sample guide feature map.

[0206] Indicative, such as Figure 12 As shown, the computer device extracts features from the first sample guide image 1208 , the second sample guide image 1209 , and the third sample guide image 1210 through the feature extraction network 1204 to obtain a first sample guide feature map 1211 , a second sample guide feature map 1212 , and a third sample guide feature map 1213 .

[0207] Step 1104: Generate a sample feature vector group corresponding to the sample feature map group and a guided feature vector group corresponding to the guided feature map group through a feature extraction network. The sample feature vector group includes a first sample feature vector, a second sample feature vector, and a third sample feature vector, and the guided feature vector group includes a first guided feature vector, a second guided feature vector, and a third guided feature vector.

[0208] For each extracted feature map, the computer device performs embedding processing on the feature map through the feature extraction network to obtain the corresponding feature vector.

[0209] Indicative, such as Figure 12As shown, the computer device performs embedding processing on the extracted feature maps to obtain a first sample feature vector 1214, a second sample feature vector 1215, a third sample feature vector 1216, a first guide feature vector 1217, a second guide feature vector 1218 and a third guide feature vector 1219.

[0210] Step 1105 : Determine the sample feature loss based on the first sample feature vector, the second sample feature vector, and the third sample feature vector in the sample feature vector group.

[0211] Since the sample image group consists of three sample images containing positive and negative samples, in this embodiment, the computer device calculates the vector distance (L2 distance) between the sample feature vectors, thereby determining the triplet loss of the sample image group based on the vector distance between different sample images, and thus determining the triplet loss as the sample feature loss. The calculation formula of the triplet loss is as follows:

[0212] l tri =max(||x a -x p ||-||x a -x n ||+α,0)

[0213] Among them, x a is the sample feature vector of the first sample image (anchor), x p is the sample feature vector of the second sample image (positive), x n is the sample feature vector of the third sample image (negative), α is a constant (for example, 0.2, that is, the purpose of the triplet loss is to make the distance between the first sample image and the third sample image greater than the distance between the first sample image and the second sample image, and the distance difference is greater than α.

[0214] Indicative, such as Figure 12 As shown, the computer device determines the triplet loss of the first sample feature vector 1213 , the second sample feature vector 1215 , and the third sample feature vector 1216 as the sample feature loss 1223 .

[0215] Step 1106 : Determine a guided feature loss based on the first guided feature vector, the second guided feature vector, and the third guided feature vector in the guided feature vector group.

[0216] Because the first sample image is similar to the second sample image but not to the third sample image, the first sample guidance image is similar to the second sample guidance image but not to the third sample guidance image. Similar to the process of determining the sample feature loss, the computer device calculates a triplet loss for the sample guidance image group and uses it as the guided feature loss.

[0217] Indicative, such as Figure 12 As shown, the computer device determines the triplet loss of the first guided feature vector 1217 , the second guided feature vector 1218 , and the third guided feature vector 1219 as a guided feature loss 1224 .

[0218] Step 1107: Determine the fusion feature loss based on the sample feature vector group and the guide feature vector group.

[0219] The above steps focus on different sample attention areas independently. In this step, the computer device fuses the feature vectors in the sample feature vector group and the guide feature vector group, and determines the fusion feature loss based on the fused feature vectors, thereby achieving the effect of simultaneously focusing on multiple sample attention areas. Optionally, this step may include the following steps.

[0220] 1. Concatenate the first sample feature vector and the first guide feature vector to obtain a first fused feature vector.

[0221] The computer device splices the feature vectors corresponding to the first sample image and the first sample guide image to obtain a first fused feature vector, which can simultaneously represent image features of different attention areas in the first sample image.

[0222] Indicative, such as Figure 12 As shown, the computer device concatenates the first sample feature vector 1214 and the first guide feature vector 1217 to obtain a first fused feature vector 1220 .

[0223] 2. Concatenate the second sample feature vector and the second guide feature vector to obtain a second fused feature vector.

[0224] The computer device splices the feature vectors corresponding to the second sample image and the second sample guide image to obtain a second fused feature vector, which can simultaneously represent image features of different attention areas in the second sample image.

[0225] Indicative, such as Figure 12 As shown, the computer device concatenates the second sample feature vector 1215 and the second guide feature vector 1218 to obtain a second fused feature vector 1221.

[0226] 3. Concatenate the third sample feature vector and the third guide feature vector to obtain a third fused feature vector.

[0227] The computer device splices the feature vectors corresponding to the third sample image and the third sample guide image to obtain a third fused feature vector, which can simultaneously represent image features of different attention areas in the third sample image.

[0228] Indicative, such as Figure 12 As shown, the computer device concatenates the third sample feature vector 1216 and the third guide feature vector 1219 to obtain a third fused feature vector 1222 .

[0229] 4. Determine the triplet loss of the first fused feature vector, the second fused feature vector and the third fused feature vector as the fused feature loss.

[0230] The computer device may calculate a triplet loss between the first fused feature vector, the second fused feature vector, and the third fused feature vector, thereby determining the triplet loss as the fused feature loss. Since the first sample image and the second sample image are similar images, and the third sample image is dissimilar to the first and second sample images, the fused feature loss will continue to decrease during network training, that is, the vector distance between the first fused feature vector and the second fused feature vector will continue to decrease, while the vector distance between the first fused feature vector and the third fused feature vector will continue to decrease.

[0231] Indicative, such as Figure 12 As shown, the computer device determines the triplet loss between the first fused feature vector 1220 , the second fused feature vector 1221 , and the third fused feature vector 1222 as a fused feature loss 1225 .

[0232] Step 1108: Train the feature extraction network based on the sample feature loss, the guided feature loss, and the fusion feature loss. The sample feature loss, the guided feature loss, and the fusion feature loss are all triplet losses.

[0233] Similar to step 908 , the computer device determines the total loss in the network training process based on the sample feature loss, the guided feature loss, and the fusion feature loss, thereby training the feature extraction network based on the total loss.

[0234] In this embodiment, the computer device constructs a ternary sample image group including positive and negative samples, and adopts a method of calculating the triplet loss. Based on the sample images in the ternary sample image group, the computer device determines the sample feature loss, the guided feature loss, and the fusion feature loss, and then uses the total loss of the three to train the feature extraction network, so that the feature extraction network can learn the image features of different local areas in the sample image, which helps to improve the accuracy and comprehensiveness of feature extraction when the feature extraction network is subsequently used for feature extraction.

[0235] Please refer to Figure 13, which shows a schematic diagram of the structure of a computer device provided by an exemplary embodiment of the present application. Specifically, the computer device 1300 includes a central processing unit (CPU) 1301, a system memory 1304 including a random access memory 1302 and a read-only memory 1303, and a system bus 1305 connecting the system memory 1304 and the central processing unit 1301. The computer device 1300 may also include a basic input / output system (I / O system) 1306 that helps transmit information between various components within the computer, and a mass storage device 1307 for storing an operating system 1313, application programs 1314, and other program modules 1315.

[0236] In some embodiments, the basic input / output system 1306 may include a display 1308 for displaying information and an input device 1309, such as a mouse or keyboard, for user input. Both the display 1308 and the input device 1309 are connected to the central processing unit 1301 via an input / output controller 1310 connected to the system bus 1305. The basic input / output system 1306 may also include an input / output controller 1310 for receiving and processing input from a variety of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1310 also provides output to a display screen, printer, or other types of output devices.

[0237] The mass storage device 1307 is connected to the central processing unit 1301 via a mass storage controller (not shown) connected to the system bus 1305. The mass storage device 1307 and its associated computer-readable media provide non-volatile storage for the computer device 1300. In other words, the mass storage device 1307 may include computer-readable media (not shown) such as a hard disk or drive.

[0238] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include random access memory (RAM), read-only memory (ROM), flash memory or other solid-state storage technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, tape cassettes, magnetic tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage media are not limited to the above-mentioned ones. The above-mentioned system memory 1304 and mass storage device 1307 can be collectively referred to as memory.

[0239] The memory stores one or more programs, and the one or more programs are configured to be executed by one or more central processing units 1301. The one or more programs contain instructions for implementing the above-mentioned methods. The central processing unit 1301 executes the one or more programs to implement the methods provided by the above-mentioned various method embodiments.

[0240] According to various embodiments of the present application, the computer device 1300 may also be connected to a remote computer on a network such as the Internet for operation. That is, the computer device 1300 may be connected to a network 1312 via a network interface unit 1311 connected to the system bus 1305. Alternatively, the network interface unit 1311 may be used to connect to other types of networks or remote computer systems (not shown).

[0241] The memory also includes one or more programs, which are stored in the memory and include steps executed by a computer device in the method provided in the embodiment of the present application.

[0242] Figure 14 The following is a block diagram of an image retrieval device provided by an exemplary embodiment of the present application. The device includes:

[0243] A feature extraction module 1401 is configured to extract features from a target image to obtain a first feature map of the target image, wherein the first feature map is used to represent image features of a first attention area in the target image;

[0244] an attention guiding module 1402 configured to guide attention of the target image based on the first feature map to obtain a guiding image corresponding to the target image, wherein an attention region of the guiding image is a second attention region, and the second attention region is different from the first attention region;

[0245] The feature extraction module 1401 is further configured to extract features from the guide image to obtain a second feature map of the guide image, where the second feature map is used to represent image features of the second attention area in the guide image.

[0246] A vector generation module 1403 is configured to generate a first feature vector based on the first feature map, and to generate a second feature vector based on the second feature map;

[0247] The retrieval module 1404 is configured to perform image retrieval based on the first feature vector and the second feature vector to obtain image retrieval results, where the image retrieval results include images similar to the target image.

[0248] Optionally, the attention guiding module 1402 includes:

[0249] a first mask generating unit, configured to generate a first mask corresponding to the target image based on the first feature map, wherein the attention area represented by the first mask is the first attention area;

[0250] a second mask generating unit, configured to generate a second mask based on the first mask, wherein the attention area represented by the second mask is the second attention area;

[0251] An image generating unit is configured to generate the guide image based on the second mask and the target image.

[0252] Optionally, the first mask generating unit is configured to:

[0253] Generate an original mask based on the first feature map of each channel, where the values ​​in the original mask are continuous values;

[0254] Based on the numerical average value of the original mask, binarize each value in the original mask to obtain the first mask;

[0255] The second mask generating unit is configured to:

[0256] The values ​​in the first mask are inverted to obtain the second mask.

[0257] Optionally, the image generating unit is configured to:

[0258] Inversely mapping the values ​​in the second mask from the feature space to the image space to obtain an image mask;

[0259] The target image is subjected to mask processing using the image mask to generate the guide image.

[0260] Optionally, the retrieval module 1404 is configured to:

[0261] Concatenating the first feature vector and the second feature vector to obtain a target fused feature vector;

[0262] Determining the vector distance between the target fusion feature vector and the corresponding fusion feature vector of the image in the image library;

[0263] The image retrieval result is determined based on the vector distance.

[0264] Optionally, the device further includes:

[0265] The storage module is configured to, in response to the absence of an image in the gallery that is duplicate of the target image, associate the target image with the target fusion feature vector and store it in the gallery.

[0266] Optionally, the first feature map and the second feature map are extracted by a convolutional layer of a feature extraction network, and the first feature vector and the second feature vector are generated by a fully connected layer of the feature extraction network.

[0267] Optionally, the device further includes a training module, configured to:

[0268] Performing feature extraction on the sample image group through the feature extraction network to obtain a sample feature map group corresponding to the sample image group, wherein the sample image group includes at least two sample images, and the sample feature map group includes a sample feature map corresponding to each of the sample images;

[0269] Based on the sample feature map group, attention is guided on the sample image group to obtain a sample guidance image group corresponding to the sample image group, wherein the sample guidance image group includes a sample guidance image corresponding to each of the sample images;

[0270] Performing feature extraction on the sample guidance image group through the feature extraction network to obtain a guidance feature map group corresponding to the sample guidance image group, wherein the guidance feature map group includes a sample guidance feature map corresponding to each of the sample guidance images;

[0271] Generating, by the feature extraction network, a sample feature vector group corresponding to the sample feature map group and a guide feature vector group corresponding to the guide feature map group, wherein the sample feature vector group includes a sample feature vector of each of the sample images, and the guide feature vector group includes a guide feature vector of each of the sample guide images;

[0272] The feature extraction network is trained based on the sample feature vector group and the guide feature vector group.

[0273] Optionally, the sample image group includes a first sample image and a second sample image, and the first sample image is similar to the second sample image;

[0274] The training module is specifically used to:

[0275] determining a sample feature loss based on a first sample feature vector and a second sample feature vector in the sample feature vector group, where the first sample feature vector is a feature vector of the first sample image and the second sample feature vector is a feature vector of the second sample image;

[0276] determining a guided feature loss based on a first guided feature vector and a second guided feature vector in the guided feature vector group, wherein the first guided feature vector is a feature vector of a sample guided image corresponding to the first sample image, and the second guided feature vector is a feature vector of a sample guided image corresponding to the second sample image;

[0277] Determining a fusion feature loss based on the sample feature vector group and the guide feature vector group;

[0278] The feature extraction network is trained based on the sample feature loss, the guided feature loss, and the fusion feature loss, where the sample feature loss, the guided feature loss, and the fusion feature loss are all KL divergence losses.

[0279] Optionally, the training module is specifically used to:

[0280] concatenating the first sample feature vector and the first guide feature vector to obtain a first fused feature vector;

[0281] concatenating the second sample feature vector and the second guide feature vector to obtain a second fused feature vector;

[0282] The KL divergence loss of the first fused feature vector and the second fused feature vector is determined as the fused feature loss.

[0283] Optionally, the sample image group includes a first sample image, a second sample image, and a third sample image, the first sample image is similar to the second sample image, and the third sample image is dissimilar to both the first sample image and the second sample image;

[0284] The training module is specifically used to:

[0285] determining a sample feature loss based on a first sample feature vector, a second sample feature vector, and a third sample feature vector in the sample feature vector group, wherein the first sample feature vector is a feature vector of the first sample image, the second sample feature vector is a feature vector of the second sample image, and the third sample feature vector is a feature vector of the third sample image;

[0286] determining a guided feature loss based on a first guided feature vector, a second guided feature vector, and a third guided feature vector in the guided feature vector group, wherein the first guided feature vector is a feature vector of a sample guided image corresponding to the first sample image, the second guided feature vector is a feature vector of a sample guided image corresponding to the second sample image, and the third guided feature vector is a feature vector of a sample guided image corresponding to the third sample image;

[0287] Determining a fusion feature loss based on the sample feature vector group and the guide feature vector group;

[0288] The feature extraction network is trained based on the sample feature loss, the guided feature loss, and the fusion feature loss, where the sample feature loss, the guided feature loss, and the fusion feature loss are all triplet losses.

[0289] Optionally, the training module is specifically used to:

[0290] concatenating the first sample feature vector and the first guide feature vector to obtain a first fused feature vector;

[0291] concatenating the second sample feature vector and the second guide feature vector to obtain a second fused feature vector;

[0292] concatenating the third sample feature vector and the third guide feature vector to obtain a third fused feature vector;

[0293] A triplet loss of the first fused feature vector, the second fused feature vector, and the third fused feature vector is determined as the fused feature loss.

[0294] To sum up, in the embodiments of the present application, when performing image retrieval on a target image, features of the target image are extracted, and attention of the target image is guided based on the extracted feature map to obtain a guide image having a different attention area from the target image, thereby performing image retrieval based on the feature vectors corresponding to the target image and the guide image to obtain an image retrieval result; the scheme provided in the embodiments of the present application is adopted to realize image feature extraction of multiple attention areas, and image retrieval is performed based on the image features corresponding to different attention areas, such as performing image retrieval based on the image features of the foreground area and the background area in the target image at the same time, thereby enriching the expression dimension of image features during image retrieval, improving the attention to different local areas in the image, and thereby improving the accuracy of image retrieval.

[0295] It should be noted that the apparatus provided in the above embodiments is merely exemplified by the division of the above functional modules. In actual applications, the above functions can be distributed among different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments are based on the same concept. The implementation process is detailed in the method embodiments and will not be repeated here.

[0296] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the image retrieval method described in any of the above embodiments.

[0297] The present invention provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image retrieval method described in the above embodiment.

[0298] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0299] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An image retrieval method, characterized in that: The method comprises: Performing feature extraction on the target image to obtain a first feature map of the target image, where the first feature map is used to represent image features of a first attention area in the target image; performing attention guidance on the target image based on the first feature map to obtain a guidance image corresponding to the target image, wherein an attention region of the guidance image is a second attention region, and the second attention region is different from the first attention region; performing feature extraction on the guide image to obtain a second feature map of the guide image, wherein the second feature map is used to represent image features of the second attention area in the guide image; generating a first feature vector based on the first feature map, and generating a second feature vector based on the second feature map; An image retrieval is performed based on the first feature vector and the second feature vector to obtain an image retrieval result, wherein the image retrieval result includes similar images to the target image.

2. The method according to claim 1, characterized in that The step of guiding attention of the target image based on the first feature map to obtain a guide image corresponding to the target image includes: generating a first mask corresponding to the target image based on the first feature map, where the attention area represented by the first mask is the first attention area; generating a second mask based on the first mask, wherein the attention area represented by the second mask is the second attention area; The guidance image is generated based on the second mask and the target image.

3. The method according to claim 2, characterized in that Generating a first mask corresponding to the target image based on the first feature map includes: Generate an original mask based on the first feature map of each channel, where the values ​​in the original mask are continuous values; Based on the numerical average value of the original mask, binarize each value in the original mask to obtain the first mask; Generating a second mask based on the first mask includes: The values ​​in the first mask are inverted to obtain the second mask.

4. The method according to claim 2, characterized in that The generating the guide image based on the second mask and the target image includes: Inversely mapping the values ​​in the second mask from the feature space to the image space to obtain an image mask; The target image is subjected to mask processing using the image mask to generate the guide image.

5. The method according to any one of claims 1 to 4, characterized in that: The performing image retrieval based on the first feature vector and the second feature vector to obtain an image retrieval result includes: Concatenating the first feature vector and the second feature vector to obtain a target fused feature vector; Determining the vector distance between the target fusion feature vector and the corresponding fusion feature vector of the image in the image library; The image retrieval result is determined based on the vector distance.

6. The method according to claim 5, characterized in that After performing image retrieval based on the first feature vector and the second feature vector to obtain an image retrieval result, the method further includes: In response to the absence of an image in the image gallery that is duplicate of the target image, the target image and the target fused feature vector are associated and stored in the image gallery.

7. The method according to any one of claims 1 to 4, characterized in that: The first feature map and the second feature map are extracted by a convolutional layer of a feature extraction network, and the first feature vector and the second feature vector are generated by a fully connected layer of the feature extraction network.

8. The method according to claim 7, characterized in that The method further comprises: Performing feature extraction on the sample image group through the feature extraction network to obtain a sample feature map group corresponding to the sample image group, wherein the sample image group includes at least two sample images, and the sample feature map group includes a sample feature map corresponding to each of the sample images; Based on the sample feature map group, attention is guided on the sample image group to obtain a sample guidance image group corresponding to the sample image group, wherein the sample guidance image group includes a sample guidance image corresponding to each of the sample images; Performing feature extraction on the sample guidance image group through the feature extraction network to obtain a guidance feature map group corresponding to the sample guidance image group, wherein the guidance feature map group includes a sample guidance feature map corresponding to each of the sample guidance images; Generating, by the feature extraction network, a sample feature vector group corresponding to the sample feature map group and a guide feature vector group corresponding to the guide feature map group, wherein the sample feature vector group includes a sample feature vector of each of the sample images, and the guide feature vector group includes a guide feature vector of each of the sample guide images; The feature extraction network is trained based on the sample feature vector group and the guide feature vector group.

9. The method according to claim 8, characterized in that The sample image group includes a first sample image and a second sample image, and the first sample image is similar to the second sample image; The step of training the feature extraction network based on the sample feature vector group and the guide feature vector group includes: determining a sample feature loss based on a first sample feature vector and a second sample feature vector in the sample feature vector group, where the first sample feature vector is a feature vector of the first sample image and the second sample feature vector is a feature vector of the second sample image; determining a guided feature loss based on a first guided feature vector and a second guided feature vector in the guided feature vector group, wherein the first guided feature vector is a feature vector of a sample guided image corresponding to the first sample image, and the second guided feature vector is a feature vector of a sample guided image corresponding to the second sample image; Determining a fusion feature loss based on the sample feature vector group and the guide feature vector group; The feature extraction network is trained based on the sample feature loss, the guided feature loss, and the fusion feature loss, where the sample feature loss, the guided feature loss, and the fusion feature loss are all KL divergence losses.

10. The method according to claim 9, characterized in that The determining of the fusion feature loss based on the sample feature vector group and the guide feature vector group includes: concatenating the first sample feature vector and the first guide feature vector to obtain a first fused feature vector; concatenating the second sample feature vector and the second guide feature vector to obtain a second fused feature vector; The KL divergence loss of the first fused feature vector and the second fused feature vector is determined as the fused feature loss.

11. The method according to claim 8, characterized in that The sample image group includes a first sample image, a second sample image, and a third sample image, the first sample image is similar to the second sample image, and the third sample image is dissimilar to both the first sample image and the second sample image; The step of training the feature extraction network based on the sample feature vector group and the guide feature vector group includes: determining a sample feature loss based on a first sample feature vector, a second sample feature vector, and a third sample feature vector in the sample feature vector group, wherein the first sample feature vector is a feature vector of the first sample image, the second sample feature vector is a feature vector of the second sample image, and the third sample feature vector is a feature vector of the third sample image; determining a guided feature loss based on a first guided feature vector, a second guided feature vector, and a third guided feature vector in the guided feature vector group, wherein the first guided feature vector is a feature vector of a sample guided image corresponding to the first sample image, the second guided feature vector is a feature vector of a sample guided image corresponding to the second sample image, and the third guided feature vector is a feature vector of a sample guided image corresponding to the third sample image; Determining a fusion feature loss based on the sample feature vector group and the guide feature vector group; The feature extraction network is trained based on the sample feature loss, the guided feature loss, and the fusion feature loss, where the sample feature loss, the guided feature loss, and the fusion feature loss are all triplet losses.

12. The method according to claim 11, characterized in that The determining of the fusion feature loss based on the sample feature vector group and the guide feature vector group includes: concatenating the first sample feature vector and the first guide feature vector to obtain a first fused feature vector; concatenating the second sample feature vector and the second guide feature vector to obtain a second fused feature vector; concatenating the third sample feature vector and the third guide feature vector to obtain a third fused feature vector; A triplet loss of the first fused feature vector, the second fused feature vector, and the third fused feature vector is determined as the fused feature loss.

13. An image retrieval device, characterized in that: The device comprises: a feature extraction module, configured to extract features from a target image to obtain a first feature map of the target image, wherein the first feature map is used to represent image features of a first attention area in the target image; an attention guiding module, configured to guide the attention of the target image based on the first feature map to obtain a guiding image corresponding to the target image, wherein the attention area of ​​the guiding image is a second attention area, and the second attention area is different from the first attention area; The feature extraction module is further configured to perform feature extraction on the guide image to obtain a second feature map of the guide image, wherein the second feature map is used to represent image features of the second attention area in the guide image; a vector generation module, configured to generate a first feature vector based on the first feature map, and generate a second feature vector based on the second feature map; The retrieval module is configured to perform image retrieval based on the first feature vector and the second feature vector to obtain image retrieval results, wherein the image retrieval results include similar images to the target image.

14. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the image retrieval method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that The readable storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the image retrieval method according to any one of claims 1 to 12.

Images