Image retrieval method and related device

Abstract

Embodiments of the present application disclose a kind of image retrieval method and related equipment, method includes: obtaining image to be inquired, determine the target type of image to be inquired, select the region of interest of image to be inquired;The region of interest is input to feature extraction network, and first key point set is obtained;According to the target type, target in pre-set base is filtered, and P second key point set is obtained;The pre-set base includes Q second key point set, each key point set corresponds to a target, Q is greater than or equal to P positive integer, P is positive integer;The first key point set, P second key point set is input to feature matching network, and P group matching relationship is obtained;According to the P group matching relationship, the similarity of image to be inquired and P second key point set corresponding image is obtained, and according to similarity, the image corresponding to P second key point set is demonstrated. Efficient image retrieval can be realized by using the embodiments of the present application.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image retrieval method and related equipment. BACKGROUND

[0002] In the prior art, image retrieval, also known as image search, is a technology of converting a query image into a vector feature and returning an image with the highest similarity to the query image through a similarity search engine. With the rapid development of multimedia technology, the amount of image data grows exponentially, and the problem of how to quickly and effectively retrieve images needs to be solved. SUMMARY

[0003] Embodiments of the present application provide an image retrieval method and related equipment, which can achieve efficient image retrieval.

[0004] In a first aspect, embodiments of the present application provide an image retrieval method, which comprises:

[0005] obtaining a query image;

[0006] extracting a region of interest of the query image;

[0007] inputting the region of interest into a feature extraction network to obtain a first key point set;

[0008] determining a target category type of the query image, and filtering key points in a preset base library according to the target category type to obtain P second key point sets, each key point set having the same category type as the target category type; the preset base library includes Q second key point sets, each key point set corresponding to an image, Q being a positive integer greater than or equal to P, and P being a positive integer;

[0009] inputting the first key point set and the P second key point sets into a feature matching network to obtain P groups of matching relationships;

[0010] determining the similarity between the query image and the images corresponding to the P second key point sets according to the P groups of matching relationships, and displaying the images corresponding to the P second key point sets according to the similarity.

[0011] In a second aspect, embodiments of the present application provide an image retrieval device, which comprises an obtaining unit, an extracting unit, an inputting unit, a determining unit and a displaying unit, wherein:

[0012] The obtaining unit is configured to obtain a query image.

[0013] The extracting unit is configured to extract a region of interest of the query image.

[0014] The input unit is configured to input the region of interest into a feature extraction network to obtain a first key point set;

[0015] The determination unit is configured to determine a target category type of the image to be queried, and filter key points in a preset base library according to the target category type to obtain P second key point sets, each key point set having a same category type as the target category type; the preset base library includes Q second key point sets, each key point set corresponding to an image, Q being a positive integer greater than or equal to P, and P being a positive integer;

[0016] The input unit is further configured to input the first key point set and the P second key point sets into a feature matching network to obtain P groups of matching relationships.

[0017] The display unit is configured to determine a similarity between the image to be queried and images corresponding to the P second key point sets according to the P groups of matching relationships, and display the images corresponding to the P second key point sets according to the similarity.

[0018] In a third aspect, an embodiment of the present application provides an electronic device, including a processor, a memory, a communication interface and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for performing the steps in the first aspect of the present application.

[0019] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program for electronic data exchange, and the computer program causes a computer to perform some or all of the steps described in the first aspect of the present application.

[0020] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a non-transitory computer readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform some or all of the steps described in the first aspect of the present application. The computer program product can be a software installation package.

[0021] By implementing the embodiments of the present application, the following beneficial effects are achieved:

[0022] It can be seen that the image retrieval method and related device described in the embodiments of the present application acquire a to-be-queried image, extract a region of interest of the to-be-queried image, input the region of interest into a feature extraction network to obtain a first key point set, determine a target category type of the to-be-queried image, and filter key points in a preset base library according to the target category type to obtain P second key point sets, each of which has the same category type as the target category type; the preset base library includes Q second key point sets, each of which corresponds to an image, Q is a positive integer greater than or equal to P, and P is a positive integer. The first key point set and the P second key point sets are input into a feature matching network to obtain P sets of matching relationships, the similarity between the to-be-queried image and the images corresponding to the P second key point sets is determined according to the P sets of matching relationships, and the images corresponding to the P second key point sets are displayed according to the similarity. On the one hand, the region of interest of the to-be-queried image can be used to filter key points of the corresponding category in the base library, and on the other hand, only the corresponding key points are matched, which helps to improve the image retrieval efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0024] Figure 1 is a flow diagram of an image retrieval method provided by an embodiment of the present application;

[0025] Figure 2 is a flow diagram of another image retrieval method provided by an embodiment of the present application;

[0026] Figure 3 is a structural diagram of an electronic device provided by an embodiment of the present application;

[0027] Figure 4 is a function unit composition block diagram of an image retrieval device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the personnel in the technical field better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0029] The terms "first", "second", and the like in the description and in the claims of the present application and above drawings are used for distinguishing between similar objects, not necessarily described in a particular order. Also, the terms "comprise", "comprising", and the like are intended to encompass non-exclusive inclusions. For example, processes, methods, articles, or apparatuses that comprise a list of steps or elements are not limited to the listed steps or elements, but can also comprise additional steps or elements not expressly listed, or can also comprise steps or elements inherent in such processes, methods, articles, or apparatuses.

[0030] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment can be included in at least one embodiment of the application. The appearances of the phrase that an embodiment in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive of one another. As will be apparent to those of ordinary skill in the art, embodiments described herein can be combinable with other embodiments.

[0031] The electronic device described in the embodiments of the present application can include a smart phone (such as an Android phone, an iOS phone, a Windows Phone, etc.), a tablet computer, a palm computer, a driving recorder, a traffic control platform, a server, a notebook computer, a mobile Internet device (MID), or a wearable device (such as a smart watch, a Bluetooth headset), etc. The above are only examples and are not exhaustive, and include but are not limited to the above electronic devices.

[0032] The embodiments of the present application are described in detail below.

[0033] Please refer to Figure 1 , Figure 1 is a flowchart of an image retrieval method provided by an embodiment of the present application, as shown in the figure, the image retrieval method comprises:

[0034] 101, obtaining a to-be-queried image.

[0035] In the embodiments of the present application, the to-be-queried image can be an image including a target object, and the target object can include at least one of the following: a face, a fingerprint, an iris, an animal, an action, and the like, which is not limited herein.

[0036] In a specific implementation, the to-be-queried image can be loaded from a memory of an electronic device or a cloud.

[0037] 102, extracting a region of interest of the to-be-queried image.

[0038] In a specific implementation, the region of interest of the to-be-searched image can be extracted to obtain a region of interest, for example, the region of interest can be a target region where a target is located. The region of interest can be one or more, and the to-be-searched image is selected and the region of interest is framed.

[0039] 103. inputting the region of interest into a feature extraction network to obtain a first key point set.

[0040] The feature extraction network can include a neural network model, which can include at least one of a convolutional neural network model, a fully connected neural network model, a recurrent neural network model, and the like, without limitation. That is, the region of interest of the query image is input into the feature extraction network to extract the key points of the region of interest of the target object.

[0041] In an embodiment of the present application, the feature extraction network can use a residual convolutional network as a backbone network, while extracting the coordinates and descriptors of the key points on the image. The network finally outputs a dense descriptor feature map (each pixel point corresponds to a descriptor), and two confidence Maps. The two confidence Maps estimate the confidence of the key point position (the reliability of the key point position) and estimate the confidence of the descriptor (the reliability of the key point descriptor). Finally, the key points are taken from the positions where the responses of the two Maps are maximized to select local key points with better expression ability and robustness. After the key points are selected, the coordinates and descriptors of the key points are input into a pre-set graph convolutional neural network to obtain the matching relationship between the image key points. The designed graph convolutional neural network is a GNN based on an attention mechanism, which simulates human feature matching. For example, it can try to select matching key points by browsing two images back and forth, and check back and forth, so as to improve the accuracy of key point matching. After matching the key points on the two images, the number of finally matched key point pairs is used as the matching similarity of the two images. The more matching point pairs, the more similar the two images are.

[0042] 104. determining a target category type of the to-be-searched image, and filtering key points in a pre-set base library according to the target category type to obtain P second key point sets, each key point set having the same category type as the target category type; the pre-set base library includes Q second key point sets, each key point set corresponding to an image, Q being a positive integer greater than or equal to P, and P being a positive integer.

[0043] The target category type can be a category type of the target object, for example, the category type can be a pedestrian, a tree, a small dog, a car, or the like. The preset database can include Q second key point sets, each key point set corresponding to an image, Q being a positive integer greater than or equal to P, P being a positive integer. Before implementing the embodiment of the present application, images containing target objects can be uploaded, and then a structured algorithm is used to perform target detection and recognition on these images, and key points of the target are extracted. Each database image corresponds to a key point set, and then the key point features (key point positions and descriptors) of the target are stored in the preset database according to the target category. In a specific implementation, the target category of the query image can be used for preliminary filtering to obtain key point features of the target with the same category in the database.

[0044] 105. inputting the first key point set and the P second key point sets into a feature matching network to obtain P groups of matching relationships.

[0045] The feature matching network is used to implement a feature matching function. The key point features of the region of interest of the query image and the key point features of the target with the same category in the database can be respectively input into the feature matching network to obtain the matching relationship between the key points of the two images.

[0046] 106. determining the similarity between the image to be queried and the P second key point set corresponding images according to the P groups of matching relationships, and displaying the P second key point set corresponding images according to the similarity.

[0047] In a specific implementation, the number of matched key point pairs is used as the matching similarity of the two images, the database images are sorted, and then the top K database images with the highest similarity are returned, K being a positive integer. Each group of the P groups of matching relationships can include the number of matched key point pairs, and the K images are displayed in descending order of the number.

[0048] The embodiment of the present application is based on an image retrieval method of a region of interest. A deep learning algorithm is used to extract key points of a region of interest of a query image and key points on a database image. These key points contain fine-grained features of the image. The key points are matched to obtain the similarity between the images, thereby improving the accuracy of image retrieval. In addition, the feature region of interest in the image is focused on, and the target with the same region of interest in the database is quickly found. The technology can also be applied to image retrieval tasks based on pedestrian and commodity regions of interest.

[0049] Optionally, the method can further include the following steps:

[0050] S1. obtaining a first sample image, the first sample image corresponding to an initial label;

[0051] S2, affine transform the first sample image through an affine transform matrix, and perform enhancement processing on the image after affine transform to obtain a second sample image;

[0052] S3, input the first sample image and the second sample image into a feature extraction network respectively to obtain two groups of outputs, the two groups of outputs being a first confidence and a first descriptor feature map of the first sample image, and a second confidence and a second descriptor feature map of the second sample image respectively;

[0053] S4, determine the coordinates and descriptors of the corresponding key points in the two groups of groups according to the initial label and the radiometric change matrix, calculate a total loss function according to the coordinates and descriptors of the corresponding key points, and update the network parameters of the feature extraction network by using the loss function;

[0054] S5, perform steps S2-S4 by using the feature extraction network with updated network parameters until the feature extraction network reaches a stable condition, perform initial key point label elimination according to the initial label and the confidence of the output of the feature extraction network, and obtain an updated initial label;

[0055] S6, perform steps S2-S4 by using the feature extraction network after reaching the stable condition, realize the use of the updated key points to replace the initial label to optimize the network, and end the training until the training is completed.

[0056] In the embodiment of the application, the feature extraction network can use a residual convolutional network as a backbone network, and finally outputs a dense descriptor feature map (each pixel point corresponds to a descriptor) and two confidence Maps. The two confidence Maps estimate the confidence of the key point position (the reliability of the key point position) and the confidence of the descriptor (the reliability of the key point descriptor) respectively. Finally, the key points are taken from the positions where the responses of the two Maps are maximized. This method simultaneously extracts the coordinates and descriptors of the key points on the image, and can select local key points with better expression ability and robustness.

[0057] In a specific implementation, in order to make the robustness of the extracted feature points better and avoid the problem of difficult training caused by random initialization, a scale invariant feature extraction transform (SIFT) algorithm is used to extract the key points of the image as the initial label of the image. Then, a training image is randomly selected and denoted as P1. The image P1 is subjected to affine transform, and the affine transform matrix is denoted as M. Then, data enhancement (including local or global enhancement of brightness, saturation, etc.) is performed on the image after affine transform, and the image is denoted as P2.

[0058] Then, the images P1 and P2 can be respectively input into the designed feature extraction network to obtain two groups of outputs O1 and O2 (including two confidence maps and descriptor feature maps) respectively, and then according to the initial labels and the image transformation relationship M, the coordinates and descriptors of the corresponding key points in the two groups of outputs O1 and O2 are found, the total loss function is calculated, and the network parameters are updated. The network parameters can be updated by back propagation.

[0059] Further, steps S2, S3 and S4 can be repeated until the network tends to be stable, and then according to the initial labels of the images and the two Maps output by the network, the initial key point labels responding to the Maps less than a threshold value are removed, and the initial labels of the images are updated. The stable condition can be pre-set or defaulted by the system, for example, the stable condition can be that the recognition accuracy of the feature extraction network reaches a certain threshold value, or the stable condition can be that the training times reach a specified number.

[0060] Further, steps S2, S3 and S4 can be repeated, the updated key point labels are used to replace the initial labels, and the network is optimized and trained until the training is completed, for example, the feature extraction network meets the convergence condition, and then the training is considered to be completed, or the training times reach a certain number, and then the training is considered to be completed.

[0061] Optionally, in the step S4, the total loss function is calculated according to the coordinates and descriptors of the corresponding key points, which can include the following steps:

[0062] S41, the key point position loss function and the key point descriptor loss function are determined according to the coordinates and descriptors of the corresponding key points respectively;

[0063] S42, the total loss function is determined according to the key point position loss function and the key point descriptor loss function;

[0064] The key point position loss function is calculated by an L2 loss function to calculate the error of the coordinates of the corresponding key points, and the key point descriptor loss function is calculated by a triplet loss function to calculate the error between the descriptor of the corresponding key points and the descriptor of any other non-corresponding key points.

[0065] In the embodiment of the application, the loss function of the feature extraction network can be composed of two parts, which includes two parts of key point position loss and descriptor loss:

[0066] L = L loc + L des

[0067] L loc is the key point position loss, which reflects the similarity of the local area of the corresponding key points in the two images; and L desFor the key point descriptor loss, the similarity of the corresponding key point descriptors on the two images is reflected.

[0068] The key point position loss function is calculated by an L2 loss function to calculate the error of the coordinate positions of the corresponding key points. The key point descriptor loss function is calculated by a triplet loss function to calculate the error between the descriptor of the corresponding key point and the descriptor of any other non-corresponding key point.

[0069] In the embodiments of the present application, feature points on the query image ROI and the base library image are extracted respectively, and then the feature points are matched. The base library image is sorted through the feature point matching result to complete the image retrieval task.

[0070] Optionally, the initial label is obtained by manual marking or a deep learning method.

[0071] The step S5 of removing the initial key point label according to the initial label and the confidence of the output of the feature extraction network to obtain the updated initial label can include the following steps:

[0072] S51, determine the threshold value corresponding to the confidence of the output of the feature extraction network.

[0073] S52, remove the initial key point whose confidence response is less than the threshold value, and update the initial label of the corresponding image.

[0074] Different confidence values can correspond to different threshold values, or two confidence values can correspond to the same threshold value.

[0075] In a specific implementation, the threshold value corresponding to the confidence of the output of the feature extraction network can be determined, the initial key point whose confidence response is less than the threshold value is removed, and the initial label of the corresponding image is updated.

[0076] Optionally, the feature matching network includes a feature point encoder for integrating the positions and descriptions of the previously extracted key points using a multi-layer perception machine; and can further include the following steps:

[0077] A1, the features output by the feature point encoder are connected using the features of the graph in and between the graph, and a learnable weight is added to the edge of the graph as an attention mechanism. Through iteration of the graph network, the matching descriptor of the feature point is finally obtained.

[0078] A2, score the matching of each point according to the feature descriptor to obtain a matching score matrix.

[0079] A3, determine a distribution matrix according to the score matrix, and the distribution matrix is used to generate a matching relationship.

[0080] In the embodiments of the present application, the key points are also called feature points, which are generally represented by coordinates (x, y) and descriptors (a string of numbers). The coordinates represent the positions of the key points, and the descriptors can be understood as a string of appearances of the key points, which are integrated together to obtain a new key point representation, that is, the visual appearance and the position of the feature point are coupled.

[0081] In the embodiments of the present application, the feature matching network can include a feature point encoder, which uses a multi-layer perception to integrate the positions and descriptions of the previously extracted key points together, and then connects the feature points in and between the graphs in the form of a graph (equivalent to a priori knowledge of the flow direction of information). The graph is a single complete graph, and the nodes of the graph are each feature point in the image. The graph includes two different undirected edges: one is an “Intra-image edge” (self edge), which connects the feature points from the image; and the other is an “Inter-image edge” (cross edge), which connects the feature points of the graph to all feature points (forming the edge) of another graph. A learnable weight can be added to the edge of the graph as an attention mechanism. Through iteration of the graph network, a matching descriptor f of the feature point is finally obtained. The matching descriptor f can be understood as a string of numbers, which is similar to the feature descriptor and is dedicated to feature matching.

[0082] Further, the matching of each point is scored according to the descriptor (using the inner product of the descriptor to measure the similarity), to obtain a matching score matrix. Due to factors such as occlusion, there may be no matching, so a row and a column are added at the end of the score matrix to place the unmatched feature points. From the score matrix to the final assignment matrix, it can be regarded as an optimal transportation problem, which can be solved by using the traditional differentiable Sinkhorn algorithm. Finally, the unmatched points are discarded to obtain the assignment matrix P. For example, the graph a and the graph b have M feature points and N feature points, respectively. The matching score matrix is an M*N matrix, which is calculated by the matching descriptor above. The matching score matrix is the matching score between the feature points of the two graphs represented by the matrix.

[0083] It can be seen that the image retrieval method described in the embodiment of the application obtains a to-be-queried image, extracts a region of interest of the to-be-queried image, inputs the region of interest into a feature extraction network to obtain a first key point set, determines a target category type of the to-be-queried image, and filters key points in a preset base library according to the target category type to obtain P second key point sets, each of which has the same category type as the target category type; the preset base library includes Q second key point sets, each of which corresponds to an image, Q is a positive integer greater than or equal to P, and P is a positive integer. The first key point set and the P second key point sets are input into a feature matching network to obtain P groups of matching relationships, the similarity between the to-be-queried image and the images corresponding to the P second key point sets is determined according to the P groups of matching relationships, and the images corresponding to the P second key point sets are displayed according to the similarity. On the one hand, the region of interest of the to-be-queried image can be used to filter key points of the corresponding category in the base library, and on the other hand, only the corresponding key points are matched, which helps to improve the image retrieval efficiency.

[0084] For example, in the embodiment of the application, vehicles, non-standard vehicles, and pedestrians are all targets, and many key points can be extracted for each target, which are recorded as a set (key point set). The base library stores many key point sets extracted from various targets. When the to-be-queried image is a vehicle, the key point set corresponding to the vehicle (filtered from non-vehicle targets) is found in the base library, and is matched with the key point set extracted from the region of interest of the to-be-queried image. The meaning of "separately" is that P second key point sets are taken one by one to match with the key point set extracted from the region of interest of the image, a total of P times.

[0085] Consistent with the above Figure 1 , please refer to Figure 2 , Figure 2 is a flowchart of an image retrieval method provided by the embodiment of the application, applied to an electronic device. As shown in the figure, the image retrieval method includes the following steps.

[0086] 201, obtaining a to-be-queried image.

[0087] 202, extracting a region of interest of the to-be-queried image.

[0088] 203, inputting the region of interest into a feature extraction network to obtain a first key point set.

[0089] 204, determining a target category type of the to-be-queried image, and filtering key points in a preset base library according to the target category type to obtain P second key point sets, each of which has the same category type as the target category type; the preset base library includes Q second key point sets, each of which corresponds to an image, Q is a positive integer greater than or equal to P, and P is a positive integer.

[0090] 205、input the first key point set and the P second key point sets to a feature matching network to obtain P groups of matching relationships.

[0091] 206、sort images corresponding to the Q second key point sets according to the P groups of matching relationships.

[0092] 207、display K images with high rankings after sorting.

[0093] The specific description of steps 201-207 can be referred to the above Figure 1 The corresponding steps of the image retrieval method are not described here.

[0094] It can be seen that the image retrieval method described in the embodiment of the application obtains a query image, extracts a region of interest of the query image, inputs the region of interest to a feature extraction network to obtain a first key point set, determines a target category type of the query image, and filters key points in a preset base library according to the target category type to obtain P second key point sets, each key point set having the same category type as the target category type. The preset base library includes Q second key point sets, each key point set corresponding to an image, Q being a positive integer greater than or equal to P, P being a positive integer. The first key point set and the P second key point sets are input to a feature matching network to obtain P groups of matching relationships. The images corresponding to the Q second key point sets are sorted according to the P groups of matching relationships. K images with high rankings after sorting are displayed. On the one hand, the region of interest of the query image can be used to filter key points of the corresponding category in the base library. On the other hand, only the corresponding key points are matched, which helps to improve the image retrieval efficiency.

[0095] The above embodiment is consistent with Figure 3 , Figure 3 is a structural schematic diagram of an electronic device provided by an embodiment of the application. As shown in the figure, the electronic device includes a processor, a memory, a communication interface, and one or more programs, and is applied to an electronic device. The above one or more programs are stored in the above memory and are configured to be executed by the above processor. In the embodiment of the application, the program includes instructions for executing the following steps:

[0096] obtain a query image;

[0097] extract a region of interest of the query image;

[0098] input the region of interest to a feature extraction network to obtain a first key point set;

[0099] determine a target category type of the image to be queried, and filter key points in a preset base library according to the target category type to obtain P second key point sets, a category type of each key point set being the same as the target category type; the preset base library comprises Q second key point sets, each key point set corresponding to an image, Q being a positive integer greater than or equal to P, and P being a positive integer;

[0100] input the first key point set and the P second key point sets into a feature matching network to obtain P groups of matching relationships;

[0101] determine similarities between the image to be queried and images corresponding to the P second key point sets according to the P groups of matching relationships, and display the images corresponding to the P second key point sets according to the similarities.

[0102] Optionally, the program further comprises instructions for performing the following steps:

[0103] S1, a first sample image is obtained, the first sample image corresponding to an initial label;

[0104] S2, the first sample image is subjected to affine transformation through an affine transformation matrix, and the image after the affine transformation is subjected to enhancement processing to obtain a second sample image;

[0105] S3, the first sample image and the second sample image are respectively input into a feature extraction network to obtain two groups of outputs, the two groups of outputs being a first confidence and a first descriptor feature map of the first sample image and a second confidence and a second descriptor feature map of the second sample image respectively;

[0106] S4, coordinates and descriptors of corresponding key points in the two groups of outputs are determined according to the initial label and the affine transformation matrix, a total loss function is calculated according to the coordinates and the descriptors of the corresponding key points, and network parameters of the feature extraction network are updated by using the loss function;

[0107] S5, the feature extraction network after the network parameter update is used to perform steps S2-S4 until the feature extraction network reaches a stable condition, an initial key point label is removed according to the initial label and a confidence of an output of the feature extraction network, and an updated initial label is obtained;

[0108] S6, the feature extraction network after the stable condition is reached is used to perform steps S2-S4, an updated key point is used to replace the initial label to optimize the network until the training is completed.

[0109] Optionally, in the aspect of calculating the total loss function according to the coordinates and the descriptors of the corresponding key points, the program comprises instructions for performing the following steps:

[0110] determining a key point position loss function and a key point descriptor loss function according to the coordinates and descriptors of the corresponding key points respectively;

[0111] determining the total loss function according to the key point position loss function and the key point descriptor loss function;

[0112] The key point position loss function is calculated by an L2 loss function to calculate the error of the coordinate position of the corresponding key point, and the key point descriptor loss function is calculated by a triplet loss function to calculate the error between the descriptor of the corresponding key point and the descriptor of any other non-corresponding key point.

[0113] Optionally, the initial label is obtained by manual marking or a deep learning method.

[0114] In the aspect of performing initial key point label elimination according to the confidence of the output of the feature extraction network to obtain an updated initial label, the above program includes instructions for performing the following steps:

[0115] determining a threshold corresponding to the confidence of the output of the feature extraction network;

[0116] eliminating the initial key point whose response is less than the threshold, and updating the initial label of the corresponding image.

[0117] Optionally, the feature matching network includes a feature point encoder for integrating the position and description of the previously extracted key points together using a multi-layer perception machine; the above program further includes instructions for performing the following steps:

[0118] connecting the features output by the feature point encoder using the form of a graph to connect the intra-graph and inter-graph feature points, and adding a learnable weight on the edge of the graph as an attention mechanism, and finally obtaining a matching descriptor of the feature points through iteration of the graph network;

[0119] scoring the matching of each point according to the feature descriptor to obtain a matching score matrix;

[0120] determining an assignment matrix according to the score matrix, the assignment matrix being used to generate a matching relationship.

[0121] It can be seen that the electronic device described in the embodiments of the present application acquires a to-be-queried image, extracts a region of interest of the to-be-queried image, inputs the region of interest into a feature extraction network to obtain a first key point set, determines a target category type of the to-be-queried image, and filters key points in a preset base library according to the target category type to obtain P second key point sets, each of which has the same category type as the target category type; the preset base library includes Q second key point sets, each of which corresponds to an image, Q is a positive integer greater than or equal to P, and P is a positive integer; the first key point set and the P second key point sets are input into a feature matching network to obtain P sets of matching relationships, the similarity between the to-be-queried image and the images corresponding to the P second key point sets is determined according to the P sets of matching relationships, and the images corresponding to the P second key point sets are displayed according to the similarity. On the one hand, the key points of the corresponding category in the base library can be filtered out through the region of interest of the to-be-queried image, and on the other hand, only the corresponding key points are matched, which helps to improve the image retrieval efficiency.

[0122] Figure 4 is a functional unit composition block diagram of the image retrieval device 400 involved in the embodiments of the present application. The image retrieval device 400 is applied to an electronic device, and the device 400 includes an acquisition unit 401, an extraction unit 402, an input unit 403, a determination unit 404, and a display unit 405, wherein

[0123] The acquisition unit 401 is configured to acquire a to-be-queried image.

[0124] The extraction unit 402 is configured to extract a region of interest of the to-be-queried image.

[0125] The input unit 403 is configured to input the region of interest into a feature extraction network to obtain a first key point set.

[0126] The determination unit 404 is configured to determine a target category type of the to-be-queried image, and filter key points in a preset base library according to the target category type to obtain P second key point sets, each of which has the same category type as the target category type; the preset base library includes Q second key point sets, each of which corresponds to an image, Q is a positive integer greater than or equal to P, and P is a positive integer.

[0127] The input unit 403 is further configured to input the first key point set and the P second key point sets into a feature matching network to obtain P sets of matching relationships.

[0128] The display unit 405 is configured to determine the similarity between the to-be-queried image and the images corresponding to the P second key point sets according to the P sets of matching relationships, and display the images corresponding to the P second key point sets according to the similarity.

[0129] Optionally, the device 400 is also specific for:

[0130] S1, obtaining a first sample image, the first sample image corresponding to an initial label;

[0131] S2, performing affine transformation on the first sample image through an affine transformation matrix, and performing enhancement processing on the image after affine transformation to obtain a second sample image;

[0132] S3, inputting the first sample image and the second sample image into a feature extraction network respectively to obtain two groups of outputs, the two groups of outputs being a first confidence and a first descriptor feature map of the first sample image, and a second confidence and a second descriptor feature map of the second sample image respectively;

[0133] S4, determining coordinates and descriptors of corresponding key points in the two groups according to the initial label and the affine transformation matrix, calculating a total loss function according to the coordinates and descriptors of the corresponding key points, and updating network parameters of the feature extraction network using the loss function;

[0134] S5, performing steps S2-S4 using the feature extraction network with updated network parameters until the feature extraction network reaches a stable condition, performing initial key point label elimination according to the initial label and the confidence of the output of the feature extraction network to obtain an updated initial label;

[0135] S6, performing steps S2-S4 using the feature extraction network after reaching the stable condition to realize using the updated key points to replace the initial label to optimize the network until the training is completed.

[0136] Optionally, in the aspect of calculating the total loss function according to the coordinates and descriptors of the corresponding key points, the device 400 is specific for:

[0137] determining a key point position loss function and a key point descriptor loss function according to the coordinates and descriptors of the corresponding key points respectively;

[0138] determining the total loss function according to the key point position loss function and the key point descriptor loss function;

[0139] The key point position loss function calculates the error of the coordinate position of the corresponding key point by an L2 loss function; and the key point descriptor loss function calculates the error between the descriptor of the corresponding key point and the descriptor of any other non-corresponding key point by a triplet loss function.

[0140] Optionally, the initial label is obtained by manual marking or a deep learning method.

[0141] In the aspect of performing initial key point label elimination according to the confidence of the output of the feature extraction network to obtain updated initial labels, the apparatus 400 is further specifically configured to:

[0142] determine a threshold corresponding to the confidence of the output of the feature extraction network;

[0143] eliminate initial key points whose response is less than the threshold, and update the initial labels of the corresponding images.

[0144] Optionally, the feature matching network comprises a feature point encoder configured to integrate the positions and descriptions of the previously extracted key points together using a multi-layer perception machine; the apparatus 400 is further specifically configured to:

[0145] connect the features output by the feature point encoder using a graph form of intra-graph and inter-graph feature points, and add a learnable weight on the edge of the graph as an attention mechanism, and finally obtain a matching descriptor of the feature points through iteration of the graph network;

[0146] score the matching of each point according to the feature descriptor to obtain a matching score matrix;

[0147] determine an assignment matrix according to the score matrix, the assignment matrix being used to generate a matching relationship.

[0148] It can be seen that the image retrieval apparatus described in the embodiments of the present application acquires a query image, extracts a region of interest of the query image, inputs the region of interest into a feature extraction network to obtain a first key point set, determines a target category type of the query image, and filters key points in a preset base library according to the target category type to obtain P second key point sets, each of which has the same category type as the target category type; the preset base library comprises Q second key point sets, each of which corresponds to an image, Q is a positive integer greater than or equal to P, and P is a positive integer; the first key point set and the P second key point sets are input into a feature matching network to obtain P matching relationships, the similarity between the query image and the images corresponding to the P second key point sets is determined according to the P matching relationships, and the images corresponding to the P second key point sets are displayed according to the similarity; on the one hand, the region of interest of the query image can be used to filter key points of the corresponding category in the base library, and on the other hand, only the corresponding key points are matched, which helps to improve the image retrieval efficiency.

[0149] It can be understood that the functions of the program modules of the image retrieval apparatus in the embodiments can be implemented according to the methods in the method embodiments, and the specific implementation process can be referred to the related description of the method embodiments, which will not be repeated here.

[0150] The embodiment of the present application further provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, and the computer program causes a computer to execute part or all of the steps of any method described in the above method embodiments, and the computer includes an electronic device.

[0151] The embodiment of the present application further provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute part or all of the steps of any method described in the above method embodiments. The computer program product can be a software installation package, and the computer includes an electronic device.

[0152] It should be noted that, for the above-mentioned method embodiments, in order to simply describe, they are all described as a combination of a series of actions, but those skilled in the art should know that the present application is not limited to the order of the actions described, because according to the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0153] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0154] In several embodiments provided in the present application, it should be understood that the disclosed apparatus can be implemented by other means. For example, the apparatus embodiments described above are only schematic, and the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical or other forms.

[0155] The units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0156] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0157] When the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable memory. Based on this understanding, the technical solutions of the present application, essentially or the part that contributes to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a memory and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned memory includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0158] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by a program instructing relevant hardware, and the program can be stored in a computer readable memory, which can include: a flash disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0159] The embodiments of the present application are described in detail above, and the principles and implementation manners of the present application are described by applying specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea; at the same time, for those of ordinary skill in the art, according to the idea of the present application, the specific implementation manner and application range will be changed, and the above description of the embodiments should not be understood as a limitation of the present application.

Claims

1. An image retrieval method characterized by, The method comprises: acquiring a to-be-queried image; extracting a region of interest of the to-be-queried image; inputting the region of interest into a feature extraction network to obtain a first key point set; determining a target category type of the to-be-queried image, and filtering key points in a preset base library according to the target category type to obtain P second key point sets, each key point set being of the same category type as the target category type; the preset base library comprises Q second key point sets, each key point set corresponding to an image, Q being a positive integer greater than or equal to P, and P being a positive integer; inputting the first key point set and the P second key point sets into a feature matching network to obtain P groups of matching relationships; determining the similarity between the to-be-queried image and images corresponding to the P second key point sets according to the P groups of matching relationships, and displaying the images corresponding to the P second key point sets according to the similarity; wherein the method further comprises: S1, acquiring a first sample image, the first sample image corresponding to an initial label; S2, performing affine transformation on the first sample image through an affine transformation matrix, and performing enhancement processing on the image after affine transformation to obtain a second sample image; S3, inputting the first sample image and the second sample image into a feature extraction network respectively to obtain two groups of outputs, the two groups of outputs being a first confidence and a first descriptor feature map of the first sample image, and a second confidence and a second descriptor feature map of the second sample image respectively; S4, determining the coordinates and descriptors of the corresponding key points in the two groups of outputs according to the initial label and the affine transformation matrix, calculating a total loss function according to the coordinates and descriptors of the corresponding key points, and updating the network parameters of the feature extraction network by using the loss function; S5, using the feature extraction network with updated network parameters to perform steps S2-S4 until the feature extraction network reaches a stable condition, performing initial key point label elimination according to the initial label and the confidence of the output of the feature extraction network to obtain an updated initial label; S6, using the feature extraction network after reaching the stable condition to perform steps S2-S4, replacing the initial label with the updated key points to optimize the network until the training is completed.

2. The method of claim 1, wherein, The total loss function is calculated according to the coordinates and descriptors of the corresponding key points, comprising: determining a key point position loss function and a key point descriptor loss function according to the coordinates and descriptors of the corresponding key points respectively; determining the total loss function according to the key point position loss function and the key point descriptor loss function; wherein the key point position loss function calculates the error of the coordinate position of the corresponding key point by an L2 loss function; and the key point descriptor loss function calculates the error between the descriptor of the corresponding key point and the descriptor of any other non-corresponding key point by a triplet loss function.

3. The method of claim 1, wherein, The initial label is obtained by manual marking or a deep learning method. The initial key point label elimination according to the confidence of the output of the feature extraction network and the initial label is performed to obtain an updated initial label, including: determining a threshold corresponding to the confidence of the output of the feature extraction network; eliminating an initial key point whose response is less than the threshold, and updating the initial label of the corresponding image.

4. The method according to any one of claims 1 to 3, characterized in that, The feature matching network includes a feature point encoder configured to integrate the position and description of the previously extracted key points together using a multi-layer perception; the method further includes: connecting the features output by the feature point encoder using a graph form of intra-graph and inter-graph feature points, and adding a learnable weight on the edge of the graph as an attention mechanism, and finally obtaining a matching descriptor of the feature points through iteration of the graph network; scoring the matching of each point according to the feature descriptor to obtain a matching score matrix; determining an assignment matrix according to the score matrix, the assignment matrix being used to generate a matching relationship.

5. An image retrieval apparatus characterized by comprising: The device includes an acquisition unit, an extraction unit, an input unit, a determination unit, and a display unit, wherein: the acquisition unit is configured to acquire a to-be-queried image; the extraction unit is configured to extract a region of interest of the to-be-queried image; the input unit is configured to input the region of interest to a feature extraction network to obtain a first key point set; the determination unit is configured to determine a target category type of the to-be-queried image, and filter key points in a preset base library according to the target category type to obtain P second key point sets, each key point set having the same category type as the target category type; the preset base library includes Q second key point sets, each key point set corresponding to an image, Q being a positive integer greater than or equal to P, and P being a positive integer; the input unit is further configured to input the first key point set and the P second key point sets to a feature matching network to obtain P groups of matching relationships; the display unit is configured to determine a similarity between the to-be-queried image and images corresponding to the P second key point sets according to the P groups of matching relationships, and display the images corresponding to the P second key point sets according to the similarity; wherein the device is further configured to: S1, acquire a first sample image, the first sample image corresponding to an initial label; S2, perform affine transformation on the first sample image through an affine transformation matrix, and perform enhancement processing on the image after affine transformation to obtain a second sample image; S3, input the first sample image and the second sample image into a feature extraction network respectively to obtain two groups of outputs, the two groups of outputs being a first confidence and a first descriptor feature map of the first sample image, and a second confidence and a second descriptor feature map of the second sample image respectively; S4, determine the coordinates and descriptors of the corresponding key points in the two groups of outputs according to the initial label and the affine transformation matrix, calculate a total loss function according to the coordinates and descriptors of the corresponding key points, and update the network parameters of the feature extraction network using the loss function. S5, performing steps S2-S4 using the feature extraction network updated with the network parameters until the feature extraction network reaches a stable condition, performing initial keypoint label pruning according to the initial labels and the confidence of the output of the feature extraction network, to obtain updated initial labels; S6, performing steps S2-S4 using the feature extraction network after reaching the stable condition, to replace the initial labels with the updated keypoint labels to optimize the network until the training is completed.

6. The apparatus of claim 5, wherein, In the aspect of calculating the total loss function according to the coordinates and descriptors of the corresponding key points, the device is specifically configured to: determine a key point position loss function and a key point descriptor loss function according to the coordinates and descriptors of the corresponding key points, respectively; determine the total loss function according to the key point position loss function and the key point descriptor loss function; wherein the key point position loss function is calculated by an L2 loss function to calculate the error of the coordinate position of the corresponding key points; and the key point descriptor loss function is calculated by a triplet loss function to calculate the error between the descriptor of the corresponding key points and the descriptor of any other non-corresponding key points.

7. An electronic device, comprising: A computer program product comprising a processor, a memory storing one or more programs configured for execution by the processor, the programs comprising instructions for performing the steps of the method of any of claims 1-4.

8. A computer-readable storage medium, characterized in that, A computer program for electronic data interchange, wherein the computer program causes a computer to perform the method of any of claims 1-4.

Images