Image retrieval method, electronic device and related products

Abstract

The embodiment of the present application discloses an image retrieval method, an electronic device and related products, the method comprising: obtaining a target area in a target image; when the target area is an area of ​​interest, determining a first global descriptor according to an index of a target object in the area of ​​interest, and searching according to the first global descriptor to obtain a first search result set, the first search result set including N reference objects; extracting features from each of the N target objects to obtain key point local features, and aggregating to obtain N first area aggregated local features; obtaining reference key point local features of the area of ​​interest, and aggregating the reference key point local features into a second area aggregated local feature of a fixed length; comparing the second area aggregated local features with the N first area aggregated local features to obtain a comparison result, and determining K reference objects based on the comparison result. The use of the embodiment of the present application can improve the efficiency of image retrieval.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image retrieval method, electronic equipment and related products. Background Art

[0002] Image retrieval, also known as image search, involves converting a query image into vector features and using a similarity search engine to return images with the highest similarity to the query image. Common commercial image search engines include Google Image Search, Baidu Image Search, and Taobao Polaroid. However, image search currently remains inefficient, and improving its efficiency is an urgent issue. Summary of the Invention

[0003] The embodiments of the present application provide an image retrieval method and related products, which can improve image retrieval efficiency.

[0004] In a first aspect, an embodiment of the present application provides an image retrieval method, applied to an electronic device, the method comprising:

[0005] Obtain the target area in the target image;

[0006] When the target area is a region of interest, determining a first global descriptor according to an index of a target object in the region of interest, and performing a search based on the first global descriptor to obtain a first search result set, where the first search result set includes N reference objects, where N is a positive integer;

[0007] Performing feature extraction on each of the N reference objects to obtain key point local features, and aggregating them to obtain N first region aggregated local features corresponding to the region of interest of the query object;

[0008] Acquire reference key point local features of the region of interest, and aggregate the reference key point local features into a second region aggregated local feature of a fixed length;

[0009] The second region aggregated local features are compared with the N first region aggregated local features to obtain a comparison result by weighting similarity according to the region overlapping area, and K reference objects are determined based on the comparison result, and the K reference objects are used as second search results, and the N reference objects include the K reference objects.

[0010] In a second aspect, an embodiment of the present application provides an image retrieval device, which is applied to an electronic device. The device includes: an acquisition unit, a search unit, an extraction unit, an aggregation unit, and a determination unit, wherein:

[0011] The acquisition unit is used to acquire the target area in the target image;

[0012] The search unit is configured to, when the target area is a region of interest, determine a first global descriptor according to an index of a target object in the region of interest, and perform a search based on the first global descriptor to obtain a first search result set, where the first search result set includes N reference objects, where N is a positive integer;

[0013] The extraction unit is configured to perform feature extraction on each of the N reference objects to obtain key point local features, and aggregate them to obtain N first region aggregated local features corresponding to the region of interest of the query object;

[0014] The aggregation unit is used to obtain the reference key point local features of the region of interest and aggregate the reference key point local features into a second region aggregated local feature of a fixed length;

[0015] The determination unit is configured to compare the second region aggregated local features with the N first region aggregated local features to obtain a comparison result based on weighted similarity of region overlap areas, and determine K reference objects based on the comparison result, and use the K reference objects as second search results, wherein the N reference objects include the K reference objects.

[0016] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the program comprises instructions for executing the steps in the first aspect of the embodiment of the present application.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the above-mentioned computer-readable storage medium stores a computer program for electronic data exchange, wherein the above-mentioned computer program enables a computer to execute some or all of the steps described in the first aspect of the embodiment of the present application.

[0018] In a fifth aspect, embodiments of the present application provide a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is operable to cause a computer to perform some or all of the steps described in the first aspect of the embodiments of the present application. The computer program product may be a software installation package.

[0019] The implementation of the embodiments of the present application has the following beneficial effects:

[0020] It can be seen that the image retrieval method, electronic device and related products described in the embodiments of the present application are applied to electronic devices to obtain a target area in a target image. When the target area is a region of interest, a first global descriptor is determined according to the index of the target object where the region of interest is located, and a search is performed based on the first global descriptor to obtain a first search result set. The first search result set includes N reference objects, where N is a positive integer. Feature extraction is performed on each of the N reference objects to obtain key point local features, and N first region aggregated local features corresponding to the region of interest of the query object are aggregated to obtain reference key point local features of the region of interest, and the reference key point local features are obtained. The key point local features are aggregated into a second domain aggregated local feature of a fixed length, and the second region aggregated local features are compared with the N first region aggregated local features to obtain the comparison results based on the weighted similarity of the overlapping areas, and K reference objects are determined based on the comparison results, and the K reference objects are used as the second search result. The N reference objects include K reference objects. Thus, the global features and local features are extracted simultaneously, combined with the key point feature selection to reduce the number of key points and the key point feature aggregation compression dimension. At the same time, the image ROI area retrieval strategy method is used to solve the requirements for memory and processor, which helps to improve the efficiency of image retrieval based on ROI areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] Figure 1A This is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0023] Figure 1B This is a flowchart of an image retrieval method provided in an embodiment of the present application;

[0024] Figure 1C This is a schematic diagram illustrating a key point selection method provided in an embodiment of the present application;

[0025] Figure 1D This is a demonstration diagram of a learnable reordering model training provided by an embodiment of the present application;

[0026] Figure 1E This is a schematic diagram of the structure of a transformer network provided in an embodiment of the present application;

[0027] Figure 1Fis a schematic diagram illustrating a stacked transformer encoder module provided in an embodiment of the present application;

[0028] Figure 2 This is a flowchart of another image retrieval method provided in an embodiment of the present application;

[0029] Figure 3 is a structural diagram of another electronic device provided in an embodiment of the present application;

[0030] Figure 4 This is a block diagram of the functional units of an image retrieval device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish between different objects, not to describe a particular order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements, but may also include steps or elements not listed, or may include other steps or elements inherent to the process, method, product, or apparatus.

[0032] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0033] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0034] The electronic devices involved in the embodiments of the present application may be handheld devices including various image processing functions, intelligent robots, vehicle-mounted devices, wearable devices, computing devices or other processing devices connected to a wireless modem, as well as various forms of user equipment (UE), mobile stations (MS), terminal devices, etc. The electronic devices may also be servers or smart home devices.

[0035] In an embodiment of the present application, the smart home device may be at least one of the following: a refrigerator, a washing machine, an electric rice cooker, smart curtains, smart lights, a smart bed, a smart trash can, a microwave oven, an oven, a steamer, an air conditioner, a range hood, a server, a smart door, a smart window, a window door wardrobe, a smart speaker, a smart home, a smart chair, a smart clothes rack, a smart shower, a water dispenser, a water purifier, an air purifier, a doorbell, a monitoring system, a smart garage, a television, a projector, a smart dining table, a smart sofa, a massage chair, a treadmill, etc., and of course, other devices may also be included.

[0036] like Figure 1A As shown, Figure 1A This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device includes a processor, memory, a signal processor, a transceiver, a display, a speaker, a microphone, a random access memory (RAM), a camera, a sensor, and a network module. The memory, signal processor DSP, speaker, microphone, RAM, camera, sensor, and network module are connected to the processor, and the transceiver is connected to the signal processor.

[0037] Among them, the processor is the control center of the electronic device, which uses various interfaces and lines to connect the various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory and calling data stored in the memory, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. The processor can be a central processing unit (CPU), a graphics processing unit (GPU) or a network processor (NPU).

[0038] Furthermore, the processor may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor.

[0039] The memory is used to store software programs and / or modules. The processor executes the software programs and / or modules stored in the memory to execute various functional applications of the electronic device and image retrieval. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one software program required for a function, etc.; the data storage area may store data created based on the use of the electronic device, etc. In addition, the memory may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0040] The sensor includes at least one of the following: a light sensor, a gyroscope, an infrared proximity sensor, a vibration detection sensor, a pressure sensor, and the like. The light sensor, also known as an ambient light sensor, is used to detect the brightness of ambient light. The light sensor may include a photosensitive element and an analog-to-digital converter. The photosensitive element is used to convert the collected light signal into an electrical signal, and the analog-to-digital converter is used to convert the electrical signal into a digital signal. Optionally, the light sensor may further include a signal amplifier, which may amplify the electrical signal converted by the photosensitive element and output it to the analog-to-digital converter. The photosensitive element may include at least one of a photodiode, a phototransistor, a photoresistor, and a silicon photocell.

[0041] The camera may be a visible light camera (a normal viewing angle camera, a wide-angle camera), an infrared camera, or a dual camera (with a ranging function), which is not limited here.

[0042] The network module may be at least one of the following: a Bluetooth module, a wireless fidelity (Wi-Fi), etc., which is not limited here.

[0043] See also Figure 1B , Figure 1B This is a flow chart of an image retrieval method provided by an embodiment of the present application, as shown in the figure, which is applied to Figure 1A The electronic device shown in FIG. 1 , the image retrieval method includes:

[0044] 101. Acquire a target area in a target image.

[0045] In the embodiments of the present application, the target image may be an image including a target. The target may be set by the user or by the system default. The target may be at least one of the following: a face, a body, a license plate, a vehicle, etc., without limitation. The target region may be a target object or a region of interest within the target object.

[0046] 102. When the target area is a region of interest, determine a first global descriptor according to an index of a target object in the region of interest, and perform a search based on the first global descriptor to obtain a first search result set, where the first search result set includes N reference objects, where N is a positive integer.

[0047] In a specific implementation, when the target area is an area of ​​interest, the electronic device can search from the database of the electronic device according to the index of the target object where the area of ​​interest is located to obtain a first global descriptor, and search based on the first global descriptor to obtain a first search result set, the first search result set including N reference objects, where N is a positive integer.

[0048] 103. Perform feature extraction on each of the N reference objects to obtain key point local features, and aggregate them to obtain N first region aggregated local features corresponding to the region of interest of the query object.

[0049] The electronic device can perform feature extraction on each of the N reference objects, including extracting features using a multi-branch convolutional network. The specific method of aggregating features of each branch can be at least one of the following: GeM, REMAP, MAC, SPcC, etc., which are not limited here, to obtain a global descriptor feature. In a specific implementation, based on the above operation for extracting a global descriptor, multiple aggregated features are fused, usually one aggregated feature per branch, and all branches are fused to generate a discriminative descriptor. The fused feature is then whitened and reduced in dimension through PCA to serve as the first global descriptor.

[0050] Optionally, the above step 103, extracting features from each of the N reference objects, may include the following steps:

[0051] 31. Obtain a feature map of a reference object a, where the reference object a is any reference object among the N reference objects;

[0052] 32. Obtain a first feature of the feature map after channel average reduction, obtain a second feature of the feature map after global average pooling, and obtain an L2 distance of a neighborhood feature of the feature map within a preset range;

[0053] 33. Determine the standard deviation of the first feature and the second feature;

[0054] 34. Perform a weighted operation based on the standard deviation and the L2 distance of the neighborhood feature to obtain a first operation result;

[0055] 35. Select key point local features of the reference object a based on the first operation result and a threshold.

[0056] In a specific implementation, taking reference object a as an example, the reference object a is any reference object among N reference objects, and the preset range can be set by the user or by the system default. Specifically, the electronic device can obtain a feature map of the reference object a, obtain the first feature of the feature map after channel averaging reduction, obtain the second feature of the feature map after global average pooling, and obtain the L2 distance of the neighborhood features of the feature map within a preset range, determine the standard deviation of the first feature and the second feature, and perform a weighted operation based on the standard deviation and the L2 distance of the neighborhood features to obtain a first operation result. Based on the first operation result and the threshold selection, such as selecting a result within a certain range, the key point local features of the reference object a can be determined.

[0057] In the specific implementation, the key to extracting keypoint features is to select keypoints with high responsiveness (responsiveness within a preset responsiveness range) and high discrimination (discrimination within a preset discrimination range) as descriptors of local features. High responsiveness means that the features contain rich information, and features of the same area can be found in different images. High discrimination means that the features are representative areas and can distinguish different images. For example, the sticker on the upper right corner of the front window of a car is white on the top and blue on the bottom, or the rearview mirror inside the car has a red decoration. Such areas are highly responsive and highly discriminative. Most other areas of the same car are basically similar, and may only have high responsiveness but not necessarily high discrimination. In order to make the features have both characteristics, a high-responsiveness and high-discrimination attention module is used to learn keypoint features with high responsiveness and high discrimination.

[0058] Furthermore, if Figure 1C As shown in the figure, the specific operation of the high-response and high-discrimination attention module is as follows: First, the input feature map is subjected to a Conv-Bn-Relu operation with a kernel size of 1 to reduce the channel to 512. It is divided into two branches. The high-response branch performs the standard deviation of the channel mean feature after the average reduction channel (channel mean reduce) and the global mean feature after the global average pooling (global average pooling). In addition, the high-discrimination branch calculates the L2 distance of the neighborhood features within a certain range of the feature point and weighted sums them. Then, the two branches are element-wise multiplied. After the Conv operation with a kernel size of 1 is performed to reduce the channel to 1, the attention score map is obtained after softplus activation. Optionally, if it is the training stage, the attention score map can also be dot-producted with the input feature map after L2Normalization, that is, the L2 normalization result of the feature map is obtained, and then the vector dot product operation is performed with the result of the first operation to obtain an HxWx1 attention score map. After reshaping, cross-entropy loss optimization is performed. Finally, all channels at the corresponding position on the input feature map are selected according to the attention score map combined with the threshold, thereby obtaining the local features of the key points with high response and high discrimination.

[0059] In specific implementation, the larger the standard deviation, the more discrete the data and the greater the amplitude change. The high-response branch is essentially to filter out high-response edge points with large gradient changes. The standard deviation of the pixel channel feature is expressed as:

[0060]

[0061] The high discrimination branch is generated by the Gaussian function with a weight of radius 2 and the weighted sum of the L2 norm distance of the feature point, expressed as:

[0062]

[0063] Finally, the element-wise product of the two branches represents an attention score map with both high response and high discrimination.

[0064] 104. Obtain reference key point local features of the region of interest, and aggregate the reference key point local features into a first region aggregated local feature of a fixed length.

[0065] In a specific implementation, if each key point feature descriptor is stored independently in the database, it will lead to huge memory and search calculation costs. Therefore, in an embodiment of the present application, on the basis of key point feature selection, any number of key point features in a certain area are further aggregated to generate a discriminative regional local aggregate descriptor vector. The target feature map of people, vehicles and objects in the traffic scene is divided into k*k grids, and each grid corresponds to a local aggregate descriptor vector, which is responsible for the feature representation of the receptive field area of ​​the original image centered on the grid. When searching the ROI area of ​​the image, the local aggregate descriptor of the grid with the most overlap between the ROI area and the receptive field area is used as the query feature.

[0066] Optionally, the above step 104, aggregating the reference key point local features into a second region aggregated local feature of a fixed length, may include the following steps:

[0067] 41. Obtain a local aggregate descriptor matrix corresponding to the local features of the reference key points;

[0068] 42. Obtain a feature vector of the target area, and divide the feature vector into c segments;

[0069] 43. Obtain b key point features from the reference key point local features;

[0070] 44. Traverse each key point feature in the b key point features, determine the cluster center of each segment vector in the c segments, and obtain the cluster center vector closest to each cluster center;

[0071] 45. Determine a residual vector based on the cluster center vector and the sub-segment vector corresponding thereto;

[0072] 46. ​​Accumulate the residual vector to the local cluster descriptor matrix segment position corresponding to the residual vector, and record the number of accumulations corresponding to the corresponding cluster center;

[0073] 47. Determine an average result matrix based on the accumulated local aggregate descriptor matrix and the number of accumulations;

[0074] 48. Determine the second region aggregated local features based on the average result matrix.

[0075] In a specific implementation, the electronic device can obtain a local aggregate descriptor matrix corresponding to the local feature of the reference key point, obtain a feature vector of the target area, and divide the feature vector into c segments, where c is a positive integer, obtain b key point features from the local feature of the reference key point, where b is a positive integer, traverse each key point feature in the b key point features, determine the cluster center of each segment vector in the c segment, obtain the cluster center vector closest to each cluster center, determine the residual vector based on the cluster center vector and the corresponding sub-segment vector, accumulate the residual vector to the segment position of the local aggregate descriptor matrix corresponding to the residual vector, and record the number of accumulations corresponding to the corresponding cluster center, determine the average result matrix based on the accumulated local aggregate descriptor matrix and the number of accumulations, that is, divide the accumulated local aggregate descriptor matrix by the number of accumulations to obtain the average result matrix, and then determine the regional aggregate local feature based on the average result matrix, that is, expand and connect each row of the average result matrix to obtain a vector with a dimension of K*D, and finally, perform PCA whitening dimensionality reduction to obtain a regional local aggregate descriptor vector of fixed length.

[0076] Before step 101, keypoint features can also be extracted from all samples in the training set during the training phase. Assuming that the dimension of the feature vector D is 1024, the original feature vector is divided into M = 32 segments, so the dimension of each sub-segment subD is 32. Each segment is clustered using k-means, assuming that each segment has K = 16 clusters. After clustering, a codebook of size M * K is obtained. After obtaining the codebook, the local features of the reference keypoints can be converted into a local aggregate descriptor matrix.

[0077] Taking a traffic scene image as an example, in an embodiment of the present application, the electronic device can divide the feature map of people, vehicles, and objects in the traffic scene into a k*k grid. Each grid corresponds to a local aggregate descriptor vector, which is responsible for the feature representation of the receptive field area of ​​the original image centered on that grid. When searching the image ROI region, the local aggregate descriptor of the grid with the most overlap between the ROI region and the receptive field area is used as the query feature.

[0078] In the specific implementation, the method for generating the regional local aggregate descriptor vector is as follows: extract key point features for all samples in the training set, assuming that the dimension of the feature vector D is 1024 dimensions, divide the original feature vector dimension into M = 32 segments, then the dimension of each sub-segment subD is 32 dimensions. Use k-mean to cluster each segment separately. Assume that K=16 clusters per segment. After clustering, a codebook of size M*K is obtained. After obtaining the codebook, initialize the local aggregate descriptor matrix O of the query target. You can also obtain any number of key point features after the query target is subjected to the key point feature selection method, traverse each key point feature, find the cluster center id closest to each sub-segment vector, take the nearest cluster center vector according to the id, subtract it from the original sub-segment subD feature vector, and obtain the residual vector resD. Add the residual vector resD to the segment position of the local aggregate descriptor matrix O corresponding to the cluster center id, and record the number of accumulations of the id. Divide the local aggregate descriptor matrix O by the number of accumulations of the corresponding segment position to obtain the average result matrix. Expand and connect each row of the average result matrix to obtain a vector of dimension K*D. Finally, perform PCA dimensionality reduction to obtain a fixed-length regional local aggregate descriptor vector.

[0079] 105. Compare the second region-aggregated local features with the N first region-aggregated local features to obtain a comparison result by weighting similarity according to the region overlap area, and determine K reference objects based on the comparison result, where the N reference objects include the K reference objects, and use the K reference objects as second search results.

[0080] In a specific implementation, the electronic device can compare the regional aggregated local features with N second regional aggregated local features, i.e., similarity comparison, to obtain a comparison result, and determine K reference objects based on the comparison result, i.e., reference objects corresponding to the top K similarity comparison values, the N reference objects include K reference objects, and the K reference objects are used as the second search results.

[0081] Optionally, the above step 105, determining K reference objects based on the comparison result and using the K reference objects as the second search result, may include the following steps:

[0082] 51. Determine image indexes of K reference objects based on the comparison results;

[0083] 52. Perform a learnable reordering check based on the image index;

[0084] 53. When the verification result is consistent, directly use the K reference objects as the second search result;

[0085] 54. When the verification result is inconsistent, rearrange the K reference objects, and use the rearranged K reference objects as the second search result.

[0086] In a specific implementation, the electronic device can determine the image indexes of K reference objects based on the comparison results, and then perform a learnable reordering check based on the image indexes. When the check results are consistent, the K reference objects are directly used as the second search results. When the check results are inconsistent, the K reference objects are rearranged, and the rearranged K reference objects are used as the second search results. The second search results can also be displayed.

[0087] In the specific implementation, reordering (also called query expansion) is a commonly used technique in image retrieval. Due to changes in lighting, posture, viewing angle and occlusion, the positive sample image may not be in the topK similar features, but in the retrieval subset of the topK similar features. In this way, the topK similar features obtained by the original query can be processed to obtain correlation information, generate new extended query features, and use the new query features to reorder the original query results to improve the recall rate and accuracy. In an embodiment of the present application, the original query and the previous TopK similar features can be used as input to train a Transformer model of a stacked self-attention encoder module to learn the relevant information between different topK similar items, thereby generating an optimal new extended query feature for all top-ranked related features. This method can perform reordering operations for the retrieval results of global features and key point aggregation features. In the training stage, the trained image ROI retrieval model can be used to extract features from the training data set, and a new training data set is generated from the retrieval results. Each type of data contains a query feature and a topK similar feature, and the TripletLoss loss function is used to optimize the Transformer self-attention model.

[0088] Specifically, if Figure 1DAs shown, self-attention is a mechanism used by the Transformer model to link the current input features with other input features. This correlation information can be well used to learn the weight parameters required to generate new query features. In actual training, the sum of the input features and the sorting position encoding first shares information through a multi-head attention mechanism, and then the layer normalization layer and the fully connected layer and the feedforward neural network are performed to remap the input features to the new embedding space. By stacking multiple identical encoders to increase the capacity of the model, more contextual information can be shared. Finally, in the converted embedding space, the dot product (similarity) between the query embedding feature and the other topK embedding features is used as a weight to measure the original query feature and the topK similar features (Formula 1). This query feature weighting method is used in many other reranking methods. In the embodiment of the present application, this weight is obtained by a learning method, and the weight is then dot-producted with the original topK similar features and accumulated and summed. Finally, L2 normalization is performed to obtain a new query feature (Formula 2), which is iteratively optimized by the TripletLoss loss function. The structural diagram of the reasoning process is shown in FIG. Figure 1E As shown, the stacked Transformer encoder module is as follows Figure 1F shown.

[0089]

[0090]

[0091] sim(.,.) is the dot product operation; norm(.) is the L2 norm normalization operation; w i Similarity weights can be learned, The similarity weight can be learned and the dot product of the topk similar features can be accumulated and summed.

[0092] Optionally, the above step 54, re-arranging the K reference objects, may include the following steps:

[0093] 541. Input the query feature and the features corresponding to the K reference objects into a preset learnable re-ranking model to obtain a new extended query feature. The preset learnable re-ranking model retrieves the regional clustered local features of all training sets to obtain X query features and corresponding X search results. After sorting and encoding each query feature and search result, the query features and search results are input into a stacked self-attention encoding model to obtain an extended query feature. The extended query feature and a specified loss function are used to iteratively optimize the model parameters of the stacked self-attention encoding model to obtain the extended query feature, where X is a positive integer.

[0094] 542. Search based on the new extended query feature to obtain the rearranged K reference objects.

[0095] In a specific implementation, the specified loss function can be tripletLoss loss (loss function). The electronic device can obtain regionally aggregated local features from all training sets and perform an N-to-N search using the regionally aggregated local features. This results in N query features and their corresponding N topK most similar features. Each query feature and its corresponding topK most similar features are encoded in order and fed into a stacked self-attention encoding model to generate a new extended query feature. This is then iteratively optimized using the tripletLoss loss function. The stacked self-attention encoding model that converges after optimization serves as the preset learnable reranking model. The electronic device can obtain a learnable reranking model for the regionally aggregated local features. During reranking, the query feature and the topK most similar features are directly fed into the learnable reranking model to obtain a new extended query feature. The search is repeated to obtain a new topK similar feature, which is used as the retrieval result. Similarly, a learnable reranking model for the global descriptor can also be obtained through the above steps. The new extended query feature contains correlation information between the query feature and the topK most similar features, and has richer discriminative features. Using this feature for further querying can effectively improve retrieval accuracy.

[0096] Furthermore, optionally, after the above step 101, the following steps may be further included:

[0097] A1. When the target area is a target object, determining a second global feature descriptor of the target object;

[0098] A2. Search according to the second global feature descriptor to obtain a third search result.

[0099] In a specific implementation, when the target area is a target object, the electronic device may determine a second global feature descriptor of the target object, and then search a preset database based on the second global feature descriptor to obtain a third search result, and may also display the third search result.

[0100] In a specific implementation, the embodiment of the present application provides an end-to-end global and local image retrieval algorithm and, the algorithm uses a residual convolutional network as the backbone network and adopts multi-branch joint training, and at the same time, extracts global descriptors and local descriptors. Among them, the global descriptor can be fused by a variety of feature aggregation methods to generate a discriminative descriptor, and the fused aggregated features can include at least one of the following: GeM, REMAP, MAC, SPcC, etc., which are not limited here. The fused features can be whitened and reduced in dimension by PCA as the final global descriptor; the local descriptor induces the attention module to select key point features on the dense features in order to obtain a descriptor with high responsiveness and high discrimination. Finally, since the number of key points in the image ROI area is randomly selected, in the embodiment of the present application, a key point feature aggregation algorithm is used to generate a fixed-length regional local descriptor. The algorithm realizes the core function of image retrieval as a component of the entire image search system. The overall processing flow of the system is as follows:

[0101] For example, the electronic device may be a server, and the image retrieval method in the embodiment of the present application may include the following steps:

[0102] 1. The client uploads an image containing the target object;

[0103] 2. The server downloads the image through a process and calls the structured algorithm;

[0104] 3. Structured algorithms for target detection, recognition and feature extraction;

[0105] 4. Return structured data and image search features, and store global features in the global descriptor base library;

[0106] 5. The client selects the target object or target ROI area;

[0107] 6. If it is a target object, find the corresponding global feature descriptor in the base database according to the index;

[0108] 7. Perform 1vN search and comparison on the global feature descriptor of the target object, and return the topK search results of the global features;

[0109] 8. If it is the target ROI area, find the corresponding global feature descriptor according to the index of the target object in the ROI area to perform target retrieval;

[0110] 9. Return the indexes of the top N target objects with the highest feature similarity, and temporarily extract the key point local features of the top N objects;

[0111] 10. Select the key point local features in the target ROI area and aggregate them into a fixed-length local feature descriptor;

[0112] 11. Perform local feature retrieval and comparison within the topN objects, and then select the topK images with the highest similarity.

[0113] 12. Perform learnable reordering verification on the topK images with the highest similarity, and return the topK retrieval results of local feature reordering

[0114] The search results of step 7 and step 12 may also be displayed on the client.

[0115] In practice, for example, when a large amount of feature data accumulates in a database, query performance and accuracy will gradually decline. Specifically, traffic scene images can be divided into a k*k grid, physically storing them into multiple parts according to certain rules. To find the corresponding grid, one only needs to query and compare the vector of the corresponding grid in the database. The specific description of the image ROI area retrieval method as a sub-module of the image search system (system overall process steps 8-12) is as follows: first, perform a structural analysis on the target image, extract the global feature vector, and store it in the database; select the target ROI area from the client, perform a secondary structural analysis on the original image, and extract the global feature vector and the local feature vector of the key points; call Milvus search based on the global feature vector to return the topN most similar images; perform a secondary structural analysis on the topN most similar images to extract the global features and the local features of the key points; find the corresponding grid according to the image ROI area, and then aggregate the local features of the key points in the grid according to the codebook to generate the topN local aggregated descriptor vectors; perform a 1vN brute force search and comparison within the topN object range, and return the topK local feature most similar images; perform a learnable reordering check on the key points of the topK local feature most similar images, and return the topK retrieval results of the local feature reordering.

[0116] In one possible example, when the target object is a human face, the following steps may be further included between step 101 and step 102:

[0117] B1. Obtaining an image quality evaluation value of a target object in the target image;

[0118] B2. When the image quality evaluation value is greater than a preset image quality evaluation value, execute step 102.

[0119] In the embodiment of the present application, the preset image quality evaluation value can be pre-stored in the electronic device, which can be set by the user or by the system default.

[0120] In a specific implementation, the electronic device may use at least one image quality evaluation metric to evaluate the image quality of the facial images in the facial image set, obtaining multiple facial image quality evaluation values. The image quality evaluation metric may be at least one of the following: facial deviation, facial integrity, clarity, feature point distribution density, average gradient, information entropy, signal-to-noise ratio, etc., which are not limited herein. Furthermore, the electronic device may execute step 102 when the image quality evaluation value is greater than a preset image quality evaluation value. The facial deviation is the deviation between the facial angle in the image and the facial angle of the front face, and the facial integrity is the ratio between the area of ​​the face in the image and the area of ​​the complete face.

[0121] In one possible example, the above step B1, obtaining the image quality evaluation value of the target object in the target image, may include the following steps:

[0122] B11, obtaining a target face deviation degree of a target object, a target face completeness degree of the target object, a target feature point distribution density of the target object, and a target information entropy;

[0123] B12. When the target face deviation is greater than a preset deviation and the target face integrity is greater than a preset integrity, determining a target first reference evaluation value corresponding to the target face deviation according to a preset mapping relationship between the face deviation and the first reference evaluation value;

[0124] B13. Determine a target second reference evaluation value corresponding to the target face integrity according to a preset mapping relationship between the face integrity and the second reference evaluation value;

[0125] B14. Determine a target weight pair corresponding to the target feature point distribution density according to a preset mapping relationship between feature point distribution density and weight pairs, wherein the target weight pair includes a target first weight and a target second weight, wherein the target first weight is a weight corresponding to the first reference evaluation value, and the target second weight is a weight corresponding to the second reference evaluation value;

[0126] B15. Perform a weighted operation based on the first target weight, the second target weight, the first target reference evaluation value, and the second target reference evaluation value to obtain a first reference evaluation value;

[0127] B16. Determine a first image quality evaluation value corresponding to the target feature point distribution density according to a preset mapping relationship between the feature point distribution density and the image quality evaluation value;

[0128] B17. Determine a target image quality deviation value corresponding to the target information entropy according to a preset mapping relationship between information entropy and image quality deviation value;

[0129] B18. Acquire a first shooting parameter of the target object;

[0130] B19. Determine a target optimization coefficient corresponding to the first shooting parameter according to a preset mapping relationship between the shooting parameter and the optimization coefficient;

[0131] B20. Adjust the first image quality evaluation value according to the target optimization coefficient and the target image quality deviation value to obtain a second reference evaluation value;

[0132] B21. Obtain target environment parameters corresponding to the target object;

[0133] B22. Determine a target weight coefficient pair corresponding to the target environmental parameter according to a preset mapping relationship between environmental parameters and weight coefficient pairs, where the target weight coefficient pair includes a target first weight coefficient and a target second weight coefficient, where the target first weight coefficient is the weight coefficient corresponding to the first reference evaluation value, and the target second weight coefficient is the weight coefficient corresponding to the second reference evaluation value;

[0134] B23. Perform a weighted operation based on the target first weight coefficient, the target second weight coefficient, the first reference evaluation value, and the second reference evaluation value to obtain a facial image quality evaluation value of the target object.

[0135] In the embodiment of the present application, the preset deviation and the preset completeness can be set by the user or by the system as a default. Only when both are within a certain range can facial recognition be successful. The electronic device can pre-store a mapping relationship between a preset face deviation and a first reference evaluation value, a mapping relationship between a preset face completeness and a second reference evaluation value, and a mapping relationship between a preset feature point distribution density and a weight pair. The weight pair may include a first weight and a second weight, the sum of the first weight and the second weight being 1, the first weight being the weight corresponding to the first reference evaluation value, and the second weight being the weight corresponding to the second reference evaluation value. The electronic device may also pre-store a mapping relationship between a preset feature point distribution density and an image quality evaluation value, a mapping relationship between a preset information entropy and an image quality deviation value, a mapping relationship between preset shooting parameters and optimization coefficients, and a mapping relationship between preset environmental parameters and weight coefficient pairs. The weight coefficient pair may include a first weight coefficient and a second weight coefficient, the first weight coefficient being the weight coefficient corresponding to the first reference evaluation value, the second weight coefficient being the weight coefficient corresponding to the second reference evaluation value, and the sum of the first weight coefficient and the second weight coefficient being 1.

[0136] Among them, the image quality evaluation value can range from 0 to 1, or it can also range from 0 to 100. The image quality deviation value can be a positive real number, for example, 0 to 1, or it can also be greater than 1. The optimization coefficient can range from -1 to 1, for example, the optimization coefficient can be -0.1 to 0.1. In the embodiment of the present application, the shooting parameters can be at least one of the following: exposure time, shooting mode, sensitivity ISO, white balance parameters, focal length, focus, area of ​​interest, etc., which are not limited here. The environmental parameters can be at least one of the following: ambient brightness, ambient temperature, ambient humidity, weather, atmospheric pressure, magnetic field interference intensity, etc., which are not limited here.

[0137] In a specific implementation, the electronic device can obtain the target face deviation degree of the target object, the target face completeness of the target object, the target feature point distribution density of the target object and the target information entropy, wherein the target feature point distribution density can be the ratio between the total number of feature points of the target object and the area of ​​the target object.

[0138] Furthermore, when the target face deviation is greater than the preset deviation and the target face integrity is greater than the preset integrity, the electronic device can determine the target first reference evaluation value corresponding to the target face deviation according to the mapping relationship between the preset face deviation and the first reference evaluation value, and can also determine the target second reference evaluation value corresponding to the target face integrity according to the mapping relationship between the preset face integrity and the second reference evaluation value, and determine the target weight pair corresponding to the target feature point distribution density according to the mapping relationship between the preset feature point distribution density and the weight pair, the target weight pair including the target first weight and the target second weight, the target first weight being the weight corresponding to the first reference evaluation value, and the target second weight being the weight corresponding to the second reference evaluation value. Then, a weighted operation can be performed based on the target first weight, the target second weight, the target first reference evaluation value, and the target second reference evaluation value to obtain the first reference evaluation value. The specific calculation formula is as follows:

[0139] First reference evaluation value = target first reference evaluation value * target first weight + target second reference evaluation value * target second weight

[0140] Furthermore, the quality of the image can be evaluated overall from the perspective of the face and the completeness of the face.

[0141] Furthermore, the electronic device may determine a first image quality evaluation value corresponding to the target feature point distribution density according to a preset mapping relationship between the feature point distribution density and the image quality evaluation value, and determine a target image quality deviation value corresponding to the target information entropy according to a preset mapping relationship between the information entropy and the image quality deviation value. The electronic device may determine a target image quality deviation value corresponding to the target information entropy according to a preset mapping relationship between the information entropy and the image quality deviation value. Since some noise is generated due to external (weather, light, angle, jitter, etc.) or internal (system, GPU) reasons when generating an image, and this noise has some impact on the image quality, the image quality may be adjusted to a certain extent to ensure an objective evaluation of the image quality.

[0142] Furthermore, the electronic device may also obtain a first shooting parameter of the target facial image, and determine a target optimization coefficient corresponding to the first shooting parameter according to a mapping relationship between a preset shooting parameter and an optimization coefficient. The shooting parameter setting may also have a certain impact on the image quality evaluation. Therefore, it is necessary to determine the impact component of the shooting parameter on the image quality. Finally, the first image quality evaluation value is adjusted according to the target optimization coefficient and the target image quality deviation value to obtain a second reference evaluation value, wherein the second reference evaluation value can be obtained according to the following formula:

[0143] When the image quality evaluation value is a percentage system, the specific calculation formula is as follows:

[0144] Second reference evaluation value = (first image quality evaluation value + target image quality deviation value) * (1 + target optimization coefficient)

[0145] When the image quality evaluation value is a percentage, the specific calculation formula is as follows:

[0146] Second reference evaluation value = first image quality evaluation value * (1 + target image quality deviation value) * (1 + target optimization coefficient)

[0147] Furthermore, the electronic device can obtain the target environment parameters corresponding to the target object, and determine the target weight coefficient pair corresponding to the target environment parameters according to the mapping relationship between the preset environment parameters and the weight coefficient pair. The target weight coefficient pair includes a target first weight coefficient and a target second weight coefficient. The target first weight coefficient is the weight coefficient corresponding to the first reference evaluation value, and the target second weight coefficient is the weight coefficient corresponding to the second reference evaluation value. Then, a weighted operation can be performed based on the target first weight coefficient, the target second weight coefficient, the first reference evaluation value, and the second reference evaluation value to obtain the face image quality evaluation value of the target object. The specific calculation formula is as follows:

[0148] Target object's face image quality evaluation value = first reference evaluation value * target first weight coefficient + second reference evaluation value * target second weight coefficient

[0149] In this way, the image quality can be objectively evaluated by combining internal and external environmental factors, shooting setting factors, and the influence of facial angle and completeness, which helps to improve the accuracy of facial image quality evaluation.

[0150] It can be seen that the image retrieval method described in the embodiment of the present application is applied to an electronic device to obtain a target area in a target image. When the target area is a region of interest, a first global descriptor is determined according to the index of the target object where the region of interest is located, and a search is performed based on the first global descriptor to obtain a first search result set. The first search result set includes N reference objects, where N is a positive integer. Feature extraction is performed on each of the N reference objects to obtain key point local features, and N first region aggregated local features corresponding to the region of interest of the query object are aggregated to obtain reference key point local features of the region of interest, and the reference key point local features are obtained. Local features are aggregated into a second domain aggregated local feature of a fixed length, and the second region aggregated local feature is compared with the N first region local features to obtain a comparison result based on the weighted similarity of the overlapping area of ​​the regions, and K reference objects are determined based on the comparison result, and the K reference objects are used as the second search result. The N reference objects include the K reference objects. Thus, the global features and local features are extracted simultaneously, combined with the key point feature selection to reduce the number of key points and the key point feature aggregation to compress the dimension. At the same time, the image ROI region retrieval strategy method is used to solve the requirements for memory and processor, which helps to improve the efficiency of image retrieval based on the ROI region.

[0151] Compared with image retrieval in related technologies, in the embodiment of the present application, local image retrieval is performed by the user selecting an ROI area to retrieve images containing features of the ROI area or images with similar features. Due to the uncertainty of the ROI area, it is impossible to use global features for retrieval as in image retrieval. It is usually implemented by using key point feature retrieval methods, which poses a challenge to memory and processor performance. The research aims to achieve the retrieval of local images in traffic scenes without significantly increasing memory and processor performance. The research results have significant economic benefits and are of great significance to the system. In the embodiment of the present application, a method of simultaneous extraction of global features and local features is used, combined with key point feature selection to reduce the number of key points and key point feature aggregation to compress dimensions. At the same time, the requirements for memory and processor are solved through the strategic method of image ROI area retrieval.

[0152] With the above Figure 1B For details on the embodiments shown, please refer to Figure 2 , Figure 2This is a flow chart of an image retrieval method provided in an embodiment of the present application, which is applied to Figure 1A The electronic device shown in FIG. 1 , the image retrieval method includes:

[0153] 201. Acquire a target area in a target image.

[0154] 202. When the target area is a region of interest, determine a first global descriptor according to an index of a target object in the region of interest, and perform a search based on the first global descriptor to obtain a first search result set, where the first search result set includes N reference objects, where N is a positive integer.

[0155] 203. Perform feature extraction on each of the N reference objects to obtain key point local features, and aggregate to obtain N first region aggregated local features corresponding to the region of interest of the query object.

[0156] 204 : Acquire reference key point local features of the region of interest, and aggregate the reference key point local features into a second region aggregated local feature of a fixed length.

[0157] 205. Compare the aggregated local features of the second region with the N local features of the first region to obtain a comparison result by weighting similarity according to the overlapping area of ​​the regions, and determine K reference objects based on the comparison result, and use the K reference objects as second search results, where the N reference objects include the K reference objects.

[0158] 206. When the target area is a target object, determine a second global feature descriptor of the target object.

[0159] 207. Search according to the second global feature descriptor to obtain a third search result.

[0160] The detailed description of the above steps 201 to 207 can refer to the above Figure 1B The corresponding steps of the described image retrieval method will not be repeated here.

[0161] It can be seen that the image retrieval method described in the embodiment of the present application is applied to electronic devices. When the target area is the area of ​​interest, the global features and local features are extracted simultaneously, and the key point feature selection is combined to reduce the number of key points and the key point feature aggregation compression dimension. At the same time, the strategy method of image ROI area retrieval is used to solve the requirements for memory and processor, which helps to improve the efficiency of image retrieval based on the ROI area. When the target area is the target object, the search is directly based on the global features.

[0162] In accordance with the above embodiment, please refer to Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. As shown in the figure, the device includes a processor, a memory, a communication interface, and one or more programs. The one or more programs are stored in the memory and configured to be executed by the processor. In the embodiment of the present application, the program includes instructions for executing the following steps:

[0163] Obtain the target area in the target image;

[0164] When the target area is a region of interest, determining a first global descriptor according to an index of a target object in the region of interest, and performing a search based on the first global descriptor to obtain a first search result set, where the first search result set includes N reference objects, where N is a positive integer;

[0165] Performing feature extraction on each of the N reference objects to obtain key point local features, and aggregating them to obtain N first region aggregated local features corresponding to the region of interest of the query object;

[0166] Acquire reference key point local features of the region of interest, and aggregate the reference key point local features into a second region aggregated local feature of a fixed length;

[0167] The second region aggregated local features are compared with the N first region aggregated local features to obtain a comparison result by weighting similarity according to the region overlapping area, and K reference objects are determined based on the comparison result, and the K reference objects are used as second search results, and the N reference objects include the K reference objects.

[0168] Optionally, in the aspect of extracting features from each of the N reference objects, the program includes instructions for executing the following steps:

[0169] Obtain a feature map of a reference object a, where the reference object a is any reference object among the N reference objects;

[0170] Obtaining a first feature of the feature map after channel average reduction, obtaining a second feature of the feature map after global average pooling, and obtaining an L2 distance of a neighborhood feature of the feature map within a preset range;

[0171] determining a standard deviation of the first feature and the second feature;

[0172] Performing a weighted operation based on the standard deviation and the L2 distance of the neighborhood feature to obtain a first operation result;

[0173] Based on the first operation result and a threshold value, the key point local features of the reference object a are determined.

[0174] Optionally, in the aspect of aggregating the reference key point local features into second region aggregated local features of fixed length, the program includes instructions for performing the following steps:

[0175] Obtaining a local aggregate descriptor matrix corresponding to the local features of the reference key points;

[0176] Obtaining a feature vector of the target area, and dividing the feature vector into c segments;

[0177] Obtain b key point features from the reference key point local features;

[0178] Traversing each of the b key point features, determining the cluster center of each segment vector in the c segments, and obtaining the cluster center vector closest to each cluster center;

[0179] Determining a residual vector based on the cluster center vector and the subsegment vector corresponding thereto;

[0180] Accumulate the residual vector to the local aggregation descriptor matrix segment position corresponding to the residual vector, and record the number of accumulations corresponding to the corresponding cluster center;

[0181] Determining an average result matrix based on the accumulated local aggregated descriptor matrix and the number of accumulations;

[0182] The second region aggregated local features are determined according to the average result matrix.

[0183] Optionally, in the aspect of determining K reference objects based on the comparison result and using the K reference objects as second search results, the program includes instructions for executing the following steps:

[0184] Determine image indexes of K reference objects based on the comparison results;

[0185] Performing a learnable reordering check based on the image index;

[0186] When the verification result is consistent, directly taking the K reference objects as the second search result;

[0187] When the verification result is inconsistent, the K reference objects are rearranged, and the rearranged K reference objects are used as the second search result.

[0188] Optionally, in the aspect of rearranging the K reference objects, the program includes instructions for performing the following steps:

[0189] Inputting the query feature and the features corresponding to the K reference objects into a preset learnable re-ranking model to obtain a new extended query feature, wherein the preset learnable re-ranking model retrieves the regional clustered local features of all training sets to obtain X query features and corresponding X search results, sorting and encoding each query feature and search result, and then inputting them into a stacked self-attention encoding model to obtain an extended query feature, and using the extended query feature and a specified loss function to iteratively optimize the model parameters of the stacked self-attention encoding model to obtain the extended query feature, where X is a positive integer;

[0190] A search is performed based on the new expanded query feature to obtain the rearranged K reference objects.

[0191] Optionally, the program further includes instructions for executing the following steps:

[0192] When the target area is a target object, determining a second global feature descriptor of the target object;

[0193] A search is performed based on the second global feature descriptor to obtain a third retrieval result.

[0194] The above mainly introduces the scheme of the embodiment of the present application from the perspective of the execution process of the method side. It is understandable that, in order to realize the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily appreciate that, in combination with the units and algorithm steps of each example described in the embodiment provided herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0195] The embodiment of the present application can divide the functional units according to the above method examples. For example, each functional unit can be divided according to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0196] Figure 44 is a functional unit block diagram of an image retrieval device 400 involved in an embodiment of the present application. The device 400 is applied to an electronic device. The device 400 includes: an acquisition unit 401, a search unit 402, an extraction unit 403, an aggregation unit 404, and a determination unit 405, wherein:

[0197] The acquisition unit 401 is used to acquire a target area in a target image;

[0198] The search unit 402 is configured to, when the target area is a region of interest, determine a first global descriptor according to an index of a target object in the region of interest, and perform a search based on the first global descriptor to obtain a first search result set, where the first search result set includes N reference objects, where N is a positive integer;

[0199] The extraction unit 403 is configured to perform feature extraction on each of the N reference objects to obtain key point local features, and aggregate them to obtain N first region aggregated local features corresponding to the region of interest of the query object;

[0200] The aggregation unit 404 is configured to obtain reference key point local features of the region of interest and aggregate the reference key point local features into a second region aggregated local feature of a fixed length;

[0201] The determination unit 405 is configured to compare the second region aggregated local features with the N first region local features to obtain a comparison result based on weighted similarity of overlapping areas of the regions, and determine K reference objects based on the comparison result, and use the K reference objects as second search results, where the N reference objects include the K reference objects.

[0202] Optionally, in the aspect of extracting features from each of the N reference objects, the extraction unit 403 is specifically configured to:

[0203] Obtain a feature map of a reference object a, where the reference object a is any reference object among the N reference objects;

[0204] Obtaining a first feature of the feature map after channel average reduction, obtaining a second feature of the feature map after global average pooling, and obtaining an L2 distance of a neighborhood feature of the feature map within a preset range;

[0205] determining a standard deviation of the first feature and the second feature;

[0206] Performing a weighted operation based on the standard deviation and the L2 distance of the neighborhood feature to obtain a first operation result;

[0207] Based on the first operation result and a threshold value, the key point local features of the reference object a are determined.

[0208] Optionally, in the aspect of aggregating the reference key point local features into a second region aggregated local feature of a fixed length, the aggregating unit 404 is specifically configured to:

[0209] Obtaining a local aggregate descriptor matrix corresponding to the local features of the reference key points;

[0210] Obtaining a feature vector of the target area, and dividing the feature vector into c segments;

[0211] Obtain b key point features from the reference key point local features;

[0212] Traversing each of the b key point features, determining the cluster center of each segment vector in the c segments, and obtaining the cluster center vector closest to each cluster center;

[0213] Determining a residual vector based on the cluster center vector and the subsegment vector corresponding thereto;

[0214] Accumulate the residual vector to the local aggregation descriptor matrix segment position corresponding to the residual vector, and record the number of accumulations corresponding to the corresponding cluster center;

[0215] Determining an average result matrix based on the accumulated local aggregated descriptor matrix and the number of accumulations;

[0216] The second region aggregated local features are determined according to the average result matrix.

[0217] Optionally, in determining K reference objects based on the comparison result and using the K reference objects as second search results, the determining unit 405 is specifically configured to:

[0218] Determine image indexes of K reference objects based on the comparison results;

[0219] Performing a learnable reordering check based on the image index;

[0220] When the verification result is consistent, directly taking the K reference objects as the second search result;

[0221] When the verification result is inconsistent, the K reference objects are rearranged, and the rearranged K reference objects are used as the second search result.

[0222] Optionally, in the aspect of reordering the K reference objects, the determining unit 405 is specifically configured to:

[0223] Inputting the query feature and the features corresponding to the K reference objects into a preset learnable re-ranking model to obtain a new extended query feature, wherein the preset learnable re-ranking model retrieves the regional clustered local features of all training sets to obtain X query features and corresponding X search results, sorting and encoding each query feature and search result, and then inputting them into a stacked self-attention encoding model to obtain an extended query feature, and using the extended query feature and a specified loss function to iteratively optimize the model parameters of the stacked self-attention encoding model to obtain the extended query feature, where X is a positive integer;

[0224] A search is performed based on the new expanded query feature to obtain the rearranged K reference objects.

[0225] Optionally, the device 400 is further specifically configured to:

[0226] When the target area is a target object, determining a second global feature descriptor of the target object;

[0227] A search is performed based on the second global feature descriptor to obtain a third retrieval result.

[0228] It can be understood that the functions of each program module of the image retrieval device of this embodiment can be specifically implemented according to the method in the above method embodiment. The specific implementation process can refer to the relevant description of the above method embodiment and will not be repeated here.

[0229] An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, and the computer program enables a computer to execute part or all of the steps of any method described in the above method embodiments, and the above computer includes an electronic device.

[0230] The present application also provides a computer program product comprising a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is operable to cause a computer to perform some or all of the steps of any of the methods described in the above method embodiments. The computer program product may be a software installation package, and the computer may comprise an electronic device.

[0231] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0232] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0233] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0234] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0235] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0236] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the above-mentioned methods of each embodiment of the present application. The aforementioned memory includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0237] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program. The program can be stored in a computer-readable memory, and the memory can include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0238] The above is a detailed introduction to the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An image retrieval method, characterized in that: Applied to electronic equipment, the method comprises: Obtain a target area in a target image; When the target area is an area of ​​interest, a first global descriptor is determined according to an index of a target object in the area of ​​interest, and a search is performed according to the first global descriptor to obtain a first search result set, wherein the first search result set includes N reference objects, where N is a positive integer; Performing feature extraction on each of the N reference objects to obtain key point local features, and aggregating to obtain N first region aggregated local features corresponding to the region of interest of the query object; Acquire reference key point local features of the region of interest, and aggregate the reference key point local features into a second region aggregated local feature of a fixed length; Comparing the second region aggregated local features with the N first region aggregated local features to obtain a comparison result according to weighted similarity of region overlap areas, and determining K reference objects based on the comparison result, and using the K reference objects as second search results, wherein the N reference objects include the K reference objects; The step of extracting features from each of the N reference objects to obtain local features of key points includes: Acquire a feature map of a reference object a, where the reference object a is any reference object among the N reference objects; Obtaining a first feature of the feature map after channel average reduction, obtaining a second feature of the feature map after global average pooling, and obtaining an L2 distance of a neighborhood feature of the feature map within a preset range; determining a standard deviation of the first feature and the second feature; Performing a weighted operation based on the standard deviation and the L2 distance of the neighborhood feature to obtain a first operation result; Selecting the key point local features of the reference object a based on the first operation result combined with a threshold; The step of aggregating the reference key point local features into a second region aggregated local feature of a fixed length includes: Obtaining a local aggregated descriptor matrix corresponding to the local features of the reference key points; Obtaining a feature vector of the target area, and dividing the feature vector into c segments; Obtain b key point features from the reference key point local features; Traversing each of the b key point features, determining the cluster center of each segment vector in the c segments, and obtaining the cluster center vector closest to each cluster center; Determine a residual vector according to the cluster center vector and the sub-segment vector corresponding thereto; Accumulating the residual vector to the local aggregation descriptor matrix segment position corresponding to the residual vector, and recording the number of accumulations corresponding to the corresponding cluster center; Determine an average result matrix based on the accumulated local aggregated descriptor matrix and the number of accumulations; The second region aggregated local features are determined according to the average result matrix.

2. The method according to claim 1, characterized in that The determining K reference objects based on the comparison result, and using the K reference objects as second search results, includes: Determine image indexes of K reference objects based on the comparison results; Perform a learnable reordering check based on the image index; When the verification result is consistent, directly taking the K reference objects as the second search result; When the verification result is inconsistent, the K reference objects are rearranged, and the rearranged K reference objects are used as the second search result.

3. The method according to claim 2, characterized in that The re-arranging the K reference objects comprises: Input the feature to be queried and the features corresponding to the K reference objects into a preset learnable re-ranking model to obtain a new extended query feature, wherein the preset learnable re-ranking model is to retrieve the regional clustered local features of all training sets to obtain X query features and corresponding X search results, and after sorting and encoding each query feature and search result, send them to the stacked self-attention encoding model to obtain an extended query feature, and use the extended query feature and a specified loss function to iteratively optimize the model parameters of the stacked self-attention encoding model to obtain the extended query feature, wherein X is a positive integer; A search is performed based on the new extended query feature to obtain the rearranged K reference objects.

4. The method according to claim 1 or 2, characterized in that: The method further comprises: When the target area is a target object, determining a second global feature descriptor of the target object; A search is performed according to the second global feature descriptor to obtain a third search result.

5. An image retrieval device, characterized in that: Applied to electronic equipment, the device comprises: an acquisition unit, a search unit, an extraction unit, an aggregation unit and a determination unit, wherein: The acquisition unit is used to acquire the target area in the target image; The search unit is configured to determine a first global descriptor according to an index of a target object in the region of interest when the target region is a region of interest, and to search according to the first global descriptor to obtain a first search result set, wherein the first search result set includes N reference objects, where N is a positive integer; The extraction unit is used to extract features from each of the N reference objects to obtain key point local features, and aggregate them to obtain N first region aggregated local features; The aggregation unit is used to obtain the reference key point local features of the region of interest, and aggregate the reference key point local features into a second region aggregated local feature of a fixed length; The determining unit is used to compare the second region aggregated local features with the N first region aggregated local features to obtain a comparison result according to weighted similarity of region overlap areas, and determine K reference objects based on the comparison result, and use the K reference objects as second search results, wherein the N reference objects include the K reference objects; Wherein, in the aspect of extracting features from each of the N reference objects to obtain local features of key points, the extraction unit is specifically used for: Acquire a feature map of a reference object a, where the reference object a is any reference object among the N reference objects; Obtaining a first feature of the feature map after channel average reduction, obtaining a second feature of the feature map after global average pooling, and obtaining an L2 distance of a neighborhood feature of the feature map within a preset range; determining a standard deviation of the first feature and the second feature; Performing a weighted operation based on the standard deviation and the L2 distance of the neighborhood feature to obtain a first operation result; Selecting the key point local features of the reference object a based on the first operation result combined with a threshold; The step of aggregating the reference key point local features into a second region aggregated local feature of a fixed length includes: Obtaining a local aggregated descriptor matrix corresponding to the local features of the reference key points; Obtaining a feature vector of the target area, and dividing the feature vector into c segments; Obtain b key point features from the reference key point local features; Traversing each of the b key point features, determining the cluster center of each segment vector in the c segments, and obtaining the cluster center vector closest to each cluster center; Determine a residual vector according to the cluster center vector and the sub-segment vector corresponding thereto; Accumulating the residual vector to the local aggregation descriptor matrix segment position corresponding to the residual vector, and recording the number of accumulations corresponding to the corresponding cluster center; Determine an average result matrix based on the accumulated local aggregated descriptor matrix and the number of accumulations; The second region aggregated local features are determined according to the average result matrix.

6. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store one or more programs and is configured to be executed by the processor, wherein the program comprises instructions for executing the steps in the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: A computer program for electronic data exchange is stored, wherein the computer program enables a computer to execute the method according to any one of claims 1 to 4.

Images