Image Retrieval Method, Apparatus, Electronic Device and Storage Medium

Abstract

An embodiment of the present application provides an image retrieval method, device, electronic device and storage medium. By obtaining an image to be retrieved; using a plurality of preset sample images to perform global feature retrieval on the image to be retrieved, obtaining a plurality of candidate images that meet the preset similarity parameters, where the number of candidate images is less than the number of sample images; using the plurality of candidate images to perform local feature retrieval on the image to be retrieved to determine a target image with the highest similarity to the image to be retrieved. Since the global feature retrieval is first performed on the sample images, a plurality of candidate images with higher similarity and fewer quantities to the image to be retrieved are obtained, realizing the screening of the sample images. Then, the local feature comparison is further performed on the candidate images, which can effectively reduce the computational amount of the local feature comparison, improve the image retrieval efficiency while ensuring the retrieval accuracy, and shorten the image retrieval duration.

Description

Technical Field

[0001] Embodiments of the present application relate to the technical field of image retrieval, and in particular, to an image retrieval method, apparatus, electronic device, and storage medium. Background Art

[0002] This section aims to provide background or context for the embodiments of the present application described in the claims. The description herein is not considered prior art merely because it is included in this section.

[0003] With the rapid development of the Internet and the widespread popularity of mobile photography devices, people's lives have gradually become closely related to rich and diverse image data. As a direct form of data representation, images contain extremely rich information content, and this powerful information advantage enables images to play an extremely important role in various industries in society.

[0004] Currently, technologies for image retrieval based on visual features have been widely applied. For example, the "search by image" application in e-commerce platforms can help users quickly achieve image-based search functions. However, with the rapid development of mobile photography devices and multimedia technologies, the number of images on the Internet is rapidly expanding, and at the same time, the visual information contained in images is more complex and diverse.

[0005] Therefore, in the prior art, when retrieving a large amount of image data, as the number of images to be retrieved increases, problems such as too long retrieval time or insufficient retrieval accuracy will occur. Summary of the Invention

[0006] Embodiments of the present application provide an image retrieval method, apparatus, electronic device, and storage medium to solve the problems of too long retrieval time or insufficient retrieval accuracy when retrieving a large amount of image data.

[0007] In a first aspect, embodiments of the present application provide an image retrieval method, including:

[0008] Obtain an image to be retrieved;

[0009] Perform global feature retrieval on the image to be retrieved using a preset plurality of sample images to obtain a plurality of candidate images that meet a preset similarity parameter, where the number of the candidate images is less than the number of the sample images;

[0010] Perform local feature retrieval on the image to be retrieved using the plurality of candidate images to determine a target image with the highest similarity to the image to be retrieved.

[0011] In a possible design, performing global feature retrieval on the image to be retrieved using a preset plurality of sample images to obtain a plurality of candidate images that meet a preset similarity parameter includes:

[0012] The global features of the image to be retrieved and the sample images are respectively extracted by a neural network trained to convergence, obtaining a first global feature corresponding to the image to be retrieved and a second global feature corresponding to the sample image;

[0013] The multiple second global features are respectively compared with the first global feature for similarity, obtaining multiple global similarity values, where the multiple global similarity values correspond one-to-one with the multiple sample images;

[0014] According to the multiple global similarity values, the partial images among the multiple sample images that match the similarity parameter are determined as the alternative images.

[0015] In a possible design, the similarity parameter includes a weight threshold. According to the multiple global similarity values, determining the partial images among the multiple sample images that match the similarity parameter as the alternative images includes:

[0016] Performing normalization processing on the multiple global similarity values to obtain multiple weight values, where the multiple weight values correspond one-to-one with the multiple sample images;

[0017] The sample images corresponding to the weight values greater than the weight threshold are determined as the alternative images.

[0018] In a possible design, the global features of the image to be retrieved and the sample images are respectively extracted by a neural network trained to convergence, obtaining a first global feature corresponding to the image to be retrieved and a second global feature corresponding to the sample image, including:

[0019] The image to be retrieved and the sample images are respectively input into a pre-trained AlexNet neural network for feature extraction, obtaining a retrieved image feature and a sample image feature;

[0020] Performing feature dimensionality reduction and normalization processing on the retrieved image feature and the sample image feature to obtain the first global feature and the second global feature.

[0021] In a possible design, using the multiple alternative images to perform local feature retrieval on the image to be retrieved to determine the target image with the highest similarity to the image to be retrieved, including:

[0022] Performing local feature extraction on the image to be retrieved and the multiple alternative images respectively by the scale-invariant feature transform algorithm, obtaining multiple first local features corresponding to the image to be retrieved and multiple second local features corresponding to the sample image;

[0023] Compare the multiple first local features with the multiple second local features of each of the alternative images respectively to obtain multiple local similarity values, where the multiple local similarity values correspond to the alternative images one by one;

[0024] Determine the target image according to the multiple local similarity values, or the multiple local similarity values and the multiple global similarity values.

[0025] In a possible design, comparing the multiple first local features with the multiple second local features of each of the alternative images respectively to obtain multiple local similarity values includes:

[0026] Determine the first key local feature among the multiple first local features according to a preset weight parameter;

[0027] Compare the first key local feature with the second local feature corresponding to each alternative image respectively to obtain the local similarity value corresponding to each alternative image.

[0028] In a possible design, comparing the first key local feature with the second local feature corresponding to each alternative image respectively to obtain the local similarity value corresponding to each alternative image includes:

[0029] Based on the bag - of - words model, determine the second key local feature that matches the first key local feature among the second local features;

[0030] Calculate the similarity value between the first key local feature and the second key local feature according to the term frequency - inverse document frequency algorithm to obtain the local similarity value corresponding to each alternative image.

[0031] In a possible design, determining the target image according to the multiple local similarity values includes:

[0032] Determine the alternative image corresponding to the largest local similarity value as the target image.

[0033] In a possible design, determining the target image according to the multiple local similarity values and the multiple global similarity values includes:

[0034] Fuse the multiple local similarity values and the multiple global similarity values to obtain multiple target similarity values, where the multiple target similarity values correspond to the multiple sample images one by one;

[0035] Determine the sample image corresponding to the largest target similarity value as the target image.

[0036] In a possible design, before globally retrieving the features of the image to be retrieved using a preset multiple sample images, it further includes:

[0037] Obtain preset retrieval parameters, where the retrieval parameters are used to characterize the accuracy requirement for retrieving the image to be retrieved;

[0038] Determine the preset sample images according to the retrieval parameters.

[0039] In a second aspect, an embodiment of the present application provides an image retrieval device, including:

[0040] An acquisition module, configured to acquire an image to be retrieved;

[0041] A global retrieval module, which performs global feature retrieval on the image to be retrieved by using a plurality of preset sample images, and obtains a plurality of alternative images that meet the preset similarity parameters, where the number of the alternative images is less than the number of the sample images;

[0042] A local retrieval module, which performs local feature retrieval on the image to be retrieved by using the plurality of alternative images, and determines a target image with the highest similarity to the image to be retrieved.

[0043] In a possible design, the global retrieval module is specifically configured to:

[0044] Perform global feature extraction on the image to be retrieved and the sample images respectively through a neural network trained to convergence, and obtain a first global feature corresponding to the image to be retrieved and a second global feature corresponding to the sample images;

[0045] Compare the similarity of the plurality of second global features with the first global feature respectively to obtain a plurality of global similarity values, where the plurality of global similarity values correspond to the plurality of sample images one by one;

[0046] Determine the partial images that match the similarity parameters among the plurality of sample images as the alternative images according to the plurality of global similarity values.

[0047] In a possible design, the similarity parameter includes a weight threshold. When the global retrieval module determines the partial images that match the similarity parameters among the plurality of sample images as the alternative images according to the plurality of global similarity values, it is specifically configured to:

[0048] Perform normalization processing on the plurality of global similarity values to obtain a plurality of weight values, where the plurality of weight values correspond to the plurality of sample images one by one;

[0049] Determine the sample images corresponding to the weight values greater than the weight threshold as the alternative images.

[0050] In a possible design, when the global retrieval module extracts global features from the image to be retrieved and the sample image respectively through a neural network trained to convergence to obtain a first global feature corresponding to the image to be retrieved and a second global feature corresponding to the sample image, it specifically is used for:

[0051] Input the image to be retrieved and the sample image into the AlexNet neural network that has undergone transfer training respectively for feature extraction to obtain a retrieval image feature and a sample image feature;

[0052] Perform feature dimensionality reduction and normalization processing on the retrieval image feature and the sample image feature to obtain the first global feature and the second global feature.

[0053] In a possible design, the local retrieval module is specifically used for:

[0054] Perform local feature extraction on the image to be retrieved and the multiple alternative images respectively through the scale-invariant feature transform algorithm to obtain multiple first local features corresponding to the image to be retrieved and multiple second local features corresponding to the sample image;

[0055] Compare the multiple first local features with the multiple second local features of each alternative image respectively to obtain multiple local similarity values, where the multiple local similarity values correspond to the alternative images one by one;

[0056] Determine the target image according to the multiple local similarity values, or the multiple local similarity values and the multiple global similarity values.

[0057] In a possible design, when the local retrieval module compares the multiple first local features with the multiple second local features of each alternative image respectively to obtain multiple local similarity values, it specifically is used for:

[0058] Determine the first key local feature among the multiple first local features according to the preset weight parameters;

[0059] Compare the first key local feature with the second local feature corresponding to each alternative image respectively to obtain the local similarity value corresponding to each alternative image.

[0060] In a possible design, when the local retrieval module compares the first key local feature with the second local feature corresponding to each alternative image respectively to obtain the local similarity value corresponding to each alternative image, it specifically is used for:

[0061] Determine the second key local feature that matches the first key local feature among the second local features based on the bag-of-words model;

[0062] Calculate the similarity value between the first key local feature and the second key local feature according to the term frequency-inverse document frequency algorithm, and obtain the local similarity value corresponding to each alternative image.

[0063] In a possible design, when determining the target image according to the multiple local similarity values, the local retrieval module is specifically configured to:

[0064] Determine the alternative image corresponding to the largest local similarity value as the target image.

[0065] In a possible design, when determining the target image according to the multiple local similarity values and the multiple global similarity values, the local retrieval module is specifically configured to:

[0066] Perform feature fusion on the multiple local similarity values and the multiple global similarity values to obtain multiple target similarity values, where the multiple target similarity values correspond to the multiple sample images one by one;

[0067] Determine the sample image corresponding to the largest target similarity value as the target image.

[0068] In a possible design, before performing global feature retrieval on the to-be-retrieved image by using a preset multiple sample images, the obtaining module is further configured to:

[0069] Obtain a preset retrieval parameter, where the retrieval parameter is used to characterize the accuracy requirement for retrieving the to-be-retrieved image;

[0070] Determine the preset sample image according to the retrieval parameter.

[0071] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory and a processor, where the memory stores executable instructions of the processor; wherein, the processor is configured to execute the image retrieval method according to any one of the first aspects by executing the executable instructions.

[0072] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the image retrieval method according to any one of the first aspects is implemented.

[0073] An embodiment of the present application provides an image retrieval method, device, electronic device, and storage medium. By obtaining an image to be retrieved; using a plurality of preset sample images to perform global feature retrieval on the image to be retrieved, obtaining a plurality of candidate images that meet the preset similarity parameters, where the number of candidate images is less than the number of sample images; using the plurality of candidate images to perform local feature retrieval on the image to be retrieved, and determining a target image with the highest similarity to the image to be retrieved. Since global feature retrieval is first performed on the sample images to obtain candidate images with relatively high similarity and a small number to the image to be retrieved, screening of the sample images is achieved. Then, further local feature comparison is performed on the candidate images, which can effectively reduce the computational amount of local feature comparison, improve the image retrieval efficiency while ensuring the retrieval accuracy, and shorten the image retrieval duration. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] To more clearly illustrate the technical solutions in the embodiments of the present application or in the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following-described drawings are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0075] Figure 1 It is a specific application scenario diagram provided by an embodiment of the present application;

[0076] Figure 2 It is a flowchart of the image retrieval method provided by an embodiment of the present application;

[0077] Figure 3 It is a schematic diagram of the process of retrieving an image to be retrieved provided by this embodiment;

[0078] Figure 4 It is a flowchart of the image retrieval method provided by another embodiment of the present application;

[0079] Figure 5 For Figure 4 It is a flowchart of step S202 in the illustrated embodiment;

[0080] Figure 6 For Figure 4 It is a flowchart of step S204 in the illustrated embodiment;

[0081] Figure 7 For Figure 4 It is a flowchart of step S206 in the illustrated embodiment;

[0082] Figure 8 It is a schematic diagram of determining a target image according to the local similarity value provided by this embodiment;

[0083] Figure 9 Schematic diagram for determining a target image according to local similarity value and global similarity value provided in this embodiment;

[0084] Figure 10 Schematic structural diagram of an image retrieval device provided in an embodiment of the present application;

[0085] Figure 11 Schematic structural diagram of an electronic device provided in an embodiment of the present application.

[0086] Through the above-mentioned drawings, specific embodiments of the present application have been shown, and there will be more detailed descriptions hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. Specific embodiments

[0087] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are some, rather than all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without making creative efforts fall within the scope of protection of the present application.

[0088] Terms such as "first", "second", "third", "fourth", etc. (if any) in the specification, claims, and above-mentioned drawings of the present application are used to distinguish similar objects and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0089] First, the nouns involved in the embodiments of the present application are explained:

[0090] Global feature: A global feature refers to the overall attribute of an image. Common global features include color features, texture features, and shape features, such as intensity histograms, etc. Since they are low-level visual features at the pixel level, global features have characteristics such as good invariance, simple calculation, and intuitive representation.

[0091] Local features: Local features are features extracted from local regions of an image, including edges, corner points, lines, curves, and regions with special attributes, etc. Local image features have the characteristics of being rich in quantity in the image, having a small correlation between features, and not being affected by the disappearance of some features in the case of occlusion, and will not affect the detection and matching of other features.

[0092] Figure 1 This is a specific application scenario diagram provided by an embodiment of the present application. As Figure 1 shown, in the application scenario provided by this embodiment, the image retrieval method provided by this embodiment can be applied to an electronic device, such as a server. The server can access a sample image library storing a large number of sample images. When the server receives a retrieval instruction sent by a client, it will retrieve the to-be-retrieved image carried by the retrieval instruction in the sample database. After the retrieval is completed, one or more sample images with the highest similarity are determined from the sample database and output to the client as the result of the image retrieval, thus completing the image retrieval process.

[0093] In the prior art, in the process of retrieving an image, feature judgment is performed from two aspects: global features and local features. Among them, global features describe the information contained in the image as a whole, such as shape, texture, etc. Therefore, in the case where the image content is relatively complex, the distinguishability of global features is subject to certain limitations. On the contrary, local features use some significant feature points in the region of interest of the image to represent image information. The image descriptor based on local features can describe image information from a more subtle angle. However, a large number of feature points can often be detected in an image. How to effectively organize these feature points and establish a suitable indexing strategy has become a difficult problem in large-scale image retrieval.

[0094] To solve the above technical problems, the present invention provides an image retrieval method. By first performing global feature retrieval on the image to determine alternative images that meet the requirements, and then performing local feature retrieval on the alternative images, the process of image retrieval can improve the retrieval efficiency and reduce the retrieval duration on the premise of meeting the retrieval accuracy.

[0095] The following uses specific embodiments to elaborate in detail on the technical solution of the present application and how the technical solution of the present application solves the above technical problems. These several specific embodiments can be combined with each other. For the same or similar concepts or processes, they may not be repeated in some embodiments. The following will describe the embodiments of the present application in conjunction with the accompanying drawings.

[0096] Figure 2 This is a flowchart of the image retrieval method provided by an embodiment of the present application. Exemplarily, the image retrieval method provided by this embodiment can be executed on an electronic device such as a server, as Figure 2As shown in the figure, the image retrieval method provided in this embodiment may include:

[0097] S101. Obtain the image to be retrieved.

[0098] Exemplarily, the image to be retrieved is image data containing pixel information, and the image data contains image features that can express specific information. For example, the image to be retrieved is a photo of a commodity or an object. There are various ways to obtain the image to be retrieved. For example, receive the image data sent by a client communicating with the server, and this image data is the image to be retrieved; or receive the access address sent by a client communicating with the server, and by accessing a third-party server, obtain the image data corresponding to this access address, and this image data is the image to be retrieved. Here, the specific method of obtaining the image to be retrieved is not limited.

[0099] S102. Use a plurality of preset sample images to perform global feature retrieval on the image to be retrieved, and obtain a plurality of candidate images that meet the preset similarity parameters, where the number of candidate images is less than the number of sample images.

[0100] Specifically, the sample images are image data obtained and stored in a specific location in advance, such as a large amount of image data stored in an Internet server, or product image data stored in the server of an e-commerce platform. The server that executes the method provided in this embodiment can access the above-mentioned plurality of sample data through the network, or the above-mentioned plurality of sample data are stored in the storage medium inside the server, and the server can perform internal access to the above-mentioned plurality of sample data through a bus or other means.

[0101] Compare the image to be retrieved with the plurality of sample data in turn for global feature comparison. Exemplarily, the global features include color, shape contour, texture, etc. The sample images that have a certain similarity with the image to be retrieved at the global feature level among the plurality of sample images are determined as candidate images. Exemplarily, the image to be retrieved includes a spherical object, and the global feature is the contour feature of the object. Then, according to the feature of the spherical object included in the image to be retrieved, the sample images are retrieved in turn, and the sample images with a spherical contour feature in the sample images are screened and determined as candidate images. Exemplarily, the method for determining the global features of the sample images and the image to be retrieved can be implemented by existing image feature extraction technologies, and will not be elaborated here.

[0102] It should be noted that in the above example, the contour feature of the object is only one example of the global features. The global features can also be other features that are determined through artificial design or self-learning, and can be interpreted or not interpreted. Here, the implementation method of the global features is not specifically limited.

[0103] S103. Use multiple alternative images to perform local feature retrieval on the image to be retrieved, and determine the target image with the highest similarity to the image to be retrieved.

[0104] Specifically, the alternative image is a part of the sample image, and its own image characteristics do not change. For example, the sample images are p1 - p10, and among them, p1, p3, and p7 are confirmed as alternative images in step S102. After performing global feature retrieval on the sample images in step S102, the number of alternative images that are relatively similar at the global feature level will be less than the number of the original sample images. Further, perform local feature comparison on the alternative images to determine the images among the alternative images that are similar to the image to be retrieved at the local feature level.

[0105] Exemplarily, there are multiple methods for local feature comparison. First, local features need to be extracted from the alternative images and the image to be retrieved. Specific methods can be implemented, for example, by Scale-invariant feature transform (SIFT), SpeededUpRobustFeatures (SURF), etc. The specific parameters can be determined manually or by self-learning according to needs, and specific limitations are not imposed here. After determining their respective local features, calculate the similarity of the features respectively, and the image with the highest similarity to the image to be retrieved can be determined, that is, the target image. Among them, there are multiple specific ways to calculate the similarity, such as calculating the Manhattan distance, Euclidean distance, cosine similarity, etc., which can be set according to needs, and specific limitations are not imposed here.

[0106] Figure 3 The process schematic diagram for retrieving an image to be retrieved provided in this embodiment is as follows Figure 3 shown. By obtaining the image to be retrieved; using a preset multiple sample images to perform global feature retrieval on the image to be retrieved, obtaining multiple alternative images that meet the preset similarity parameters, where the number of alternative images is less than the number of sample images; and using the multiple alternative images to perform local feature retrieval on the image to be retrieved, determining the target image with the highest similarity to the image to be retrieved. Since the global feature retrieval is first performed on the sample images, alternative images with relatively high similarity and fewer numbers to the image to be retrieved are obtained, realizing the screening of the sample images. Then, further perform local feature comparison on the alternative images, which can effectively reduce the computational amount of local feature comparison, improve the image retrieval efficiency while ensuring the retrieval accuracy, and shorten the image retrieval duration.

[0107] Figure 4 The flowchart of the image retrieval method provided in another embodiment of this application is as follows Figure 4 shown. The image retrieval method provided in this embodiment is inFigure 2 Based on the image retrieval method provided by the illustrated embodiment, if steps S102 and S103 are further refined, the image retrieval method provided by this embodiment may include:

[0108] S201. Obtain the image to be retrieved.

[0109] S202. Respectively perform global feature extraction on the image to be retrieved and the sample image through a neural network trained to convergence, to obtain a first global feature corresponding to the image to be retrieved and a second global feature corresponding to the sample image.

[0110] Exemplarily, the neural network may be a Convolutional Neural Network (CNN), or may also be other neural networks improved based on the convolutional neural network. Among them, the convolutional neural network trained to convergence may correspond to a specific category of images to be retrieved, that is, different images to be retrieved correspond to different convolutional neural networks. Since the parameters in the convolutional neural network model are learned from a large amount of training data, these parameter values are not fixed, but will be adjusted accordingly according to the database and the actual feedback of users. Therefore, the global features extracted by the convolutional neural network can be regarded as the information extracted from the image through a large number of non-linear transformations. Among them, the intermediate data of each layer of the CNN model can express the information of a certain dimension of the image, so it has better expression ability. The global features extracted by the convolutional neural network can better adapt to the feature extraction and retrieval of complex images compared with artificial features.

[0111] In a possible design, as Figure 5 shown, S202 may include two specific implementation steps, S2021 and S2022:

[0112] S2021. Respectively input the image to be retrieved and the sample image into the AlexNet neural network after transfer learning, perform feature extraction, to obtain a retrieval image feature and a sample image feature.

[0113] The AlexNet neural network is a kind of convolutional neural network, which can better solve the problems of gradient disappearance and overfitting, and at the same time has good model generalization ability. After performing transfer learning on the AlexNet neural network, input the image to be retrieved and the sample image into the AlexNet neural network respectively, to realize the global feature extraction of the image to be retrieved and the sample image. Among them, the specific implementation process is the prior art in this field and will not be elaborated here.

[0114] S2022. Perform feature dimensionality reduction and normalization processing on the retrieval image feature and the sample image feature, to obtain a first global feature and a second global feature.

[0115] For general global feature extraction, the obtained global features are very rich, which will cause a decrease in computational efficiency. In the steps provided in this embodiment, the significance of global feature extraction lies in screening all sample images, rather than directly obtaining the final retrieval result. Therefore, during the feature extraction process, the features output by the neural network model can be dimensionally reduced to reduce the amount of data and improve computational efficiency. At the same time, in order to eliminate the influence of sudden components in the global features on the retrieval performance, the first global feature and the second global feature can be normalized to improve the stability of the first global feature and the second global feature, thereby improving the accuracy of image retrieval.

[0116] S203. Compare each of the multiple second global features with the first global feature respectively to obtain multiple global similarity values, where the multiple global similarity values correspond one-to-one to the multiple sample images.

[0117] Specifically, each second global feature corresponds to a sample image. The second global feature can be regarded as the global feature representation of the sample image, while the first global feature can be regarded as the global feature representation of the image to be retrieved. Therefore, by calculating the similarity between each second global feature and the first global feature in turn, each second global feature can obtain a similarity evaluation result, that is, a global similarity value. The global similarity value represents the similarity between the sample image and the image to be retrieved at the global feature level, and each sample image corresponds to a global similarity value.

[0118] S204. Determine the partial images that match the similarity parameter among the multiple sample images according to the multiple global similarity values as candidate images.

[0119] Specifically, the global feature is a dimension for evaluating whether different images are similar or the same. Therefore, generally speaking, the higher the global similarity value corresponding to the global feature, the higher the similarity between the corresponding sample image and the image to be retrieved. Therefore, according to the magnitude of the global similarity value, the similarity between the sample image and the image to be retrieved can be represented. Exemplarily, for example, the similarity parameter is (0.8, 1), that is, greater than 0.8 and less than 1. When the global similarity value corresponding to the sample image satisfies being greater than 0.8 and less than 1, that is, matching the similarity parameter, then the sample image can be determined as a candidate image.

[0120] In a possible design, the similarity parameter includes a weight threshold, as Figure 6 shown, S204 may include two specific implementation steps: S2041 and S2042:

[0121] S2041. Normalize the multiple global similarity values to obtain multiple weight values, where the multiple weight values correspond one-to-one to the multiple sample images.

[0122] Normalize multiple global similarity values so that all similarity values are within a fixed interval, such as (0, 1). Then, calculate the corresponding weights according to the proportion of different global similarity values among all global similarity values to obtain weight values corresponding to the sample images one by one. The specific implementation steps are the prior art in this field and will not be elaborated here.

[0123] S2042, determine the sample images corresponding to the weight values greater than the weight threshold as candidate images.

[0124] Exemplarily, the value range of the weight threshold is between (0, 1), which can be used to represent the proportion of the sample images determined as candidate images. For example, if the weight threshold is 0.5, the sample images with weight values greater than 0.5 are determined as candidate images.

[0125] In the steps of this embodiment, since the retrieval purpose of the image to be retrieved is different in different application scenarios, the corresponding retrieval accuracy is also different. The larger the weight threshold, the fewer the number of candidate images, the higher the calculation efficiency, and correspondingly, the retrieval accuracy will decrease; on the contrary, the more the number of candidate images, the higher the retrieval accuracy will be. Through the preset weight coefficient, it can be adjusted according to the specific usage scenario to improve the flexibility and applicable scenarios of this method.

[0126] S205. Respectively perform local feature extraction on the image to be retrieved and multiple candidate images through the Scale-Invariant Feature Transform (SIFT) algorithm to obtain multiple first local features corresponding to the image to be retrieved and multiple second local features corresponding to the sample images.

[0127] Exemplarily, the Scale-Invariant Feature Transform (SIFT) algorithm is a kind of description used in the field of image processing. This description has scale invariance, can detect key points in the image, is a local feature descriptor, and has good stability and invariance. Through the Scale-Invariant Feature Transform (SIFT) algorithm, the local features in the image to be retrieved and the candidate images can be detected and extracted, and the first local features and the second local features are correspondingly generated. Its specific implementation process is the prior art and will not be elaborated here.

[0128] S206. Respectively compare the multiple first local features with the multiple second local features of each candidate image to obtain multiple local similarity values, where the multiple local similarity values correspond to the candidate images one by one.

[0129] Exemplarily, after local feature extraction is performed on the image to be retrieved, multiple first local features are generated; similarly, after local feature extraction is performed on the alternative images, multiple second local features are generated corresponding to each alternative image. The first local features are used to represent the feature information at the local feature level of the image to be retrieved, and the second local features are used to represent the feature information at the local feature level of the alternative images. By calculating the similarity between the first local features and the second local features, a local similarity value can be obtained for each alternative image.

[0130] In a possible design, as Figure 7 shown, S206 may include three specific implementation steps: S2061, S2062, and S2063.

[0131] S2061. Determine the first key local features among the multiple first local features according to the preset weight parameters.

[0132] Specifically, for different images to be retrieved, they have different image features, and correspondingly, their first local features are also different. To improve the retrieval efficiency, the importance of the multiple first local features can be weighted and sorted according to the preset weight parameters. For example, the image to be retrieved has 5 first local features, namely B1 - B5. According to the preset weight parameters, the importance weights of the first local features B1, B3, and B5 are set to 0, that is, the first local features B1, B3, and B5 are not retrieved, while the importance weights of B2 and B4 are set to 1 and used as the key features. That is, B2 and B4 are the first key local features. Of course, the weight values can also be set for B1 - B5 respectively through the weight information, and the top N items with the largest weights are used as the first key local features. The method for obtaining the weight coefficients can be set manually according to the specific usage scenario or requirements, or determined by self - learning methods such as neural networks. This is not limited here.

[0133] S2062. Based on the Bag - of - words (BoW) model, determine the second key local features among the second local features that match the first key local features.

[0134] S2063. Calculate the similarity value between the first key local features and the second key local features according to the term frequency–inverse document frequency (IF - IDF) algorithm to obtain the local similarity value corresponding to each alternative image.

[0135] Precisely query the alternative images through the first key local feature based on Scale-Invariant Feature Transform (SIFT) and the second local feature, determine the effective features in the second local feature, i.e., the second key local feature, and calculate the similarity based on the first key local feature and the second key local feature according to the Term Frequency-Inverse Document Frequency (TF-IDF) algorithm, i.e., the local similarity value. Each local similarity value corresponds to an alternative image and is used to represent the similarity between the corresponding alternative image and the image to be retrieved at the local feature level.

[0136] Among them, the calculation method of the bag-of-words model based on Scale-Invariant Feature Transform and the Term Frequency-Inverse Document Frequency algorithm are existing technologies, and the implementation process thereof will not be elaborated here.

[0137] S207. Determine the target image according to multiple local similarity values, or multiple local similarity values and multiple global similarity values.

[0138] In a possible design, determining the target image according to multiple local similarity values includes: determining the alternative image corresponding to the largest local similarity value as the target image.

[0139] Figure 8 This is a schematic diagram for determining the target image according to the local similarity value provided in this embodiment. As Figure 8 shown, the global similarity values of the sample images A1 - A8 are P1 - P8; according to the global similarity values P1 - P8, the sample images with global similarity values greater than 0.6 are determined as alternative images, i.e., A1, A4, A6, A7 are alternative images. After determining the corresponding local similarity values W1 - W4 of the alternative images, a larger local similarity value means that the alternative image has a higher consistency with the image to be retrieved at the local feature level. At the same time, since the alternative images have been screened in the previous global feature retrieval stage, it can be ensured that the alternative images and the image to be retrieved have a certain similarity at the global feature level. Therefore, according to the maximum value of the local similarity value, the corresponding alternative image can be directly determined as the target image, i.e., Figure 8 A4 shown in is the target image. Since there is no need to perform fusion calculation with the global similarity value, the calculation steps are reduced and the calculation efficiency is improved.

[0140] In a possible design, determining the target image according to multiple local similarity values and multiple global similarity values includes: performing feature fusion on multiple local similarity values and multiple global similarity values to obtain multiple target similarity values, where multiple target similarity values correspond to multiple sample images one by one; determining the sample image corresponding to the largest target similarity value as the target image.

[0141] Figure 9 This is a schematic diagram for determining the target image according to the local similarity value and the global similarity value provided in this embodiment. As Figure 9As shown, the global similarity values of sample images A1 - A8 are P1 - P8; according to the global similarity values P1 - P8, the sample images with global similarity values greater than 0.6 are determined as candidate images, that is, A1, A4, A6, and A7 are candidate images. After determining the local similarity values W1 - W4 corresponding to the candidate images, the local similarity values corresponding to the candidate images are sorted according to the sample sequence of the sample images, and 0 is filled in the vacant positions to form a local similarity value array with the same length as the global similarity value array. Then, the global similarity value arrays P1, P4, P6, and P7 are multiplied point - by - point with the local similarity value arrays W1 - W4 to correspondingly obtain a target similarity value array C1, C2, C3, C4 with the same length as the global similarity value array and the local similarity value array. Among them, the sample image corresponding to the largest target similarity value C2 is the target image.

[0142] In the steps of this embodiment, by fusing the global similarity value and the local similarity value, a target similarity value that can simultaneously reflect the global features and local features of the image is obtained. Using the target similarity value to judge the target image can further improve the accuracy of image retrieval.

[0143] In a possible design, before step S201, it further includes:

[0144] S200a. Obtain a preset retrieval parameter, and the retrieval parameter is used to represent the accuracy requirement for retrieving the image to be retrieved.

[0145] S200b. Determine the preset sample images according to the retrieval parameter.

[0146] Exemplarily, the retrieval parameter can be a specific service identifier, and this service identifier corresponds to different retrieval scopes. For example, if the service identifier is A01, then only the sample images stored in the a1 database are retrieved subsequently.

[0147] Exemplarily, the retrieval parameter can also be a specific quantity, such as 100,000, that is, 100,000 images among all the sample images are retrieved subsequently.

[0148] By obtaining the retrieval parameter, the scale and quantity of the sample images can be further controlled, so that the method provided in this embodiment can better balance the retrieval duration and retrieval accuracy, and improve the usage flexibility.

[0149] Figure 10 This is the structural schematic diagram of the image retrieval device provided by the embodiment of the present application. As Figure 10 shown, the image retrieval device 3 provided in this embodiment includes:

[0150] An acquisition module 31, configured to acquire the image to be retrieved.

[0151] The global retrieval module 32 uses a plurality of preset sample images to perform global feature retrieval on the image to be retrieved, and obtains a plurality of candidate images that meet the preset similarity parameters. The number of candidate images is less than the number of sample images.

[0152] The local retrieval module 33 uses the plurality of candidate images to perform local feature retrieval on the image to be retrieved, and determines the target image with the highest similarity to the image to be retrieved.

[0153] In a possible design, the global retrieval module 32 is specifically configured to:

[0154] Perform global feature extraction on the image to be retrieved and the sample images respectively through a neural network trained to convergence, and obtain a first global feature corresponding to the image to be retrieved and a second global feature corresponding to the sample images.

[0155] Compare the plurality of second global features with the first global feature respectively to obtain a plurality of global similarity values. Among them, the plurality of global similarity values correspond to the plurality of sample images one by one.

[0156] According to the plurality of global similarity values, determine the partial images in the plurality of sample images that match the similarity parameters as candidate images.

[0157] In a possible design, the similarity parameter includes a weight threshold. When the global retrieval module 32 determines the partial images in the plurality of sample images that match the similarity parameters as candidate images according to the plurality of global similarity values, it is specifically configured to:

[0158] Perform normalization processing on the plurality of global similarity values to obtain a plurality of weight values. Among them, the plurality of weight values correspond to the plurality of sample images one by one.

[0159] Determine the sample images corresponding to the weight values greater than the weight threshold as candidate images.

[0160] In a possible design, when the global retrieval module 32 performs global feature extraction on the image to be retrieved and the sample images respectively through a neural network trained to convergence, and obtains a first global feature corresponding to the image to be retrieved and a second global feature corresponding to the sample images, it is specifically configured to:

[0161] Input the image to be retrieved and the sample images into the migrated-trained AlexNet neural network respectively for feature extraction, and obtain the retrieved image features and the sample image features.

[0162] Perform feature dimension reduction and normalization processing on the retrieved image features and the sample image features to obtain a first global feature and a second global feature.

[0163] In a possible design, the local retrieval module 33 is specifically configured to:

[0164] The Scale-Invariant Feature Transform (SIFT) algorithm is used to extract local features from the image to be retrieved and multiple candidate images respectively, obtaining multiple first local features corresponding to the image to be retrieved and multiple second local features corresponding to the sample images.

[0165] The multiple first local features are respectively compared with the multiple second local features of each candidate image to obtain multiple local similarity values, where the multiple local similarity values correspond to the candidate images one by one.

[0166] Based on the multiple local similarity values, or the multiple local similarity values and multiple global similarity values, the target image is determined.

[0167] In a possible design, when the local retrieval module 33 compares the multiple first local features with the multiple second local features of each candidate image to obtain multiple local similarity values, it is specifically configured to:

[0168] Determine the first key local feature among the multiple first local features according to the preset weight parameters.

[0169] The first key local feature is respectively compared with the second local features corresponding to each candidate image to obtain the local similarity values corresponding to each candidate image.

[0170] In a possible design, when the local retrieval module 33 compares the first key local feature with the second local features corresponding to each candidate image to obtain the local similarity values corresponding to each candidate image, it is specifically configured to:

[0171] Based on the bag-of-words model, determine the second key local features among the second local features that match the first key local feature.

[0172] Calculate the similarity value between the first key local feature and the second key local feature according to the term frequency-inverse document frequency (TF-IDF) algorithm to obtain the local similarity values corresponding to each candidate image.

[0173] In a possible design, when the local retrieval module 33 determines the target image based on the multiple local similarity values, it is specifically configured to:

[0174] Determine the candidate image corresponding to the largest local similarity value as the target image.

[0175] In a possible design, when the local retrieval module 33 determines the target image based on the multiple local similarity values and multiple global similarity values, it is specifically configured to:

[0176] Fuse the multiple local similarity values and multiple global similarity values to obtain multiple target similarity values, where the multiple target similarity values correspond to the multiple sample images one by one.

[0177] Determine the sample image corresponding to the maximum target similarity value as the target image.

[0178] In a possible design, the obtaining module 31, before globally retrieving the image to be retrieved by using a plurality of preset sample images, is further configured to:

[0179] Obtain preset retrieval parameters, where the retrieval parameters are used to characterize the accuracy requirement for retrieving the image to be retrieved.

[0180] Determine preset sample images according to the retrieval parameters.

[0181] Wherein, the obtaining module 31, the global retrieval module 32, and the local retrieval module 33 are connected in sequence. The image retrieval device 3 provided in this embodiment can execute the technical solutions of the method embodiments shown in Figures 2 - 9 Any one of them. The implementation principles and technical effects are similar, and will not be elaborated here.

[0182] Figure 11 It is a schematic structural diagram of an electronic device provided in an embodiment of the present application. As Figure 11 shown, the electronic device 4 in this embodiment may include: a processor 41 and a memory 42.

[0183] The memory 42 is used to store programs; the memory 42 may include volatile memory (English: volatile memory), such as random access memory (English: random-access memory, abbreviation: RAM), such as static random access memory (English: static random-access memory, abbreviation: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviation: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory 42 is used to store computer programs (such as application programs and functional modules for implementing the above methods), computer instructions, etc. The above computer programs, computer instructions, etc. can be stored in one or more memories 42 in a partitioned manner. And the above computer programs, computer instructions, data, etc. can be called by the processor 41.

[0184] The above computer programs, computer instructions, etc. can be stored in one or more memories 42 in a partitioned manner. And the above computer programs, computer instructions, data, etc. can be called by the processor 41.

[0185] A processor 41 for executing a computer program stored in a memory 42 to implement each step in the method according to the above embodiments.

[0186] For specific reference, please refer to the relevant descriptions in the foregoing method embodiments.

[0187] The processor 41 and the memory 42 may be of an independent structure or an integrated structure integrated together. When the processor 41 and the memory 42 are of an independent structure, the memory 42 and the processor 41 may be coupled through a bus 43.

[0188] The electronic device of this embodiment can execute the technical solutions of the method embodiments shown in Figures 2 - 9 any one of them. The implementation principles and technical effects are similar and will not be elaborated here.

[0189] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the method provided in any one of the embodiments corresponding to the present invention Figures 2 - 9 in this application.

[0190] Among them, the computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0191] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include known common knowledge or conventional technical means in the technical field not disclosed in the present application. The specification and embodiments are only to be considered as exemplary, and the true scope and spirit of the present application are pointed out by the following claims.

[0192] It should be understood that the present application is not limited to the exact structures already described and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

Claims

1. An image retrieval method, characterized in that, Including: Obtain the image to be retrieved; After respectively performing global feature extraction on the image to be retrieved and a plurality of preset sample images by a convolutional neural network trained to convergence, perform feature dimensionality reduction and normalization processing to obtain a first global feature corresponding to the image to be retrieved and a second global feature corresponding to the sample images; different types of images to be retrieved correspond to different convolutional neural networks; the plurality of sample images are determined based on retrieval parameters, and different retrieval parameters correspond to different retrieval ranges; the retrieval parameter is a service identifier; According to the first global feature and a plurality of second global features, obtain a plurality of candidate images that meet the preset similarity parameter, wherein the number of the candidate images is less than the number of the sample images; Respectively perform local feature extraction on the image to be retrieved and the plurality of candidate images to obtain a plurality of first local features corresponding to the image to be retrieved and a plurality of second local features corresponding to the candidate images; According to the weight parameters of each local feature preset, determine a first key local feature among the plurality of first local features, and based on the bag-of-words model, determine a second key local feature that matches the first key local feature among the plurality of second local features corresponding to each candidate image; According to the similarity values between the first key local feature and the second key local features corresponding to each candidate image, determine the target image with the highest similarity to the image to be retrieved.

2. The method according to claim 1, wherein The obtaining a plurality of candidate images that meet the preset similarity parameter according to the first global feature and a plurality of second global features includes: Respectively compare the plurality of second global features with the first global feature to obtain a plurality of global similarity values, wherein the plurality of global similarity values correspond to the plurality of sample images one by one; According to the plurality of global similarity values, determine the partial images among the plurality of sample images that match the similarity parameter as the candidate images.

3. The method according to claim 2, wherein The similarity parameter includes a weight threshold, and the determining the partial images among the plurality of sample images that match the similarity parameter as the candidate images according to the plurality of global similarity values includes: Perform normalization processing on the plurality of global similarity values to obtain a plurality of weight values, wherein the plurality of weight values correspond to the plurality of sample images one by one; Determine the sample images corresponding to the weight values greater than the weight threshold as the candidate images.

4. The method according to claim 2, wherein The performing global feature extraction on the image to be retrieved and the sample images by a convolutional neural network trained to convergence, and then performing feature dimensionality reduction and normalization processing to obtain a first global feature corresponding to the image to be retrieved and a second global feature corresponding to the sample images includes: Respectively input the image to be retrieved and the sample images into the AlexNet neural network after transfer training to perform feature extraction to obtain retrieval image features and sample image features; Perform feature dimensionality reduction and normalization processing on the retrieval image features and the sample image features to obtain the first global feature and the second global feature.

5. The method according to claim 1, characterized in that Performing local feature extraction on the image to be retrieved and the multiple alternative images respectively to obtain a plurality of first local features corresponding to the image to be retrieved and a plurality of second local features corresponding to the alternative images includes: Performing local feature extraction on the image to be retrieved and the multiple alternative images respectively through the Scale-Invariant Feature Transform (SIFT) algorithm to obtain a plurality of first local features corresponding to the image to be retrieved and a plurality of second local features corresponding to the alternative images.

6. The method according to any one of claims 2-4, characterized in that, Determining the target image with the highest similarity to the image to be retrieved according to the similarity values between the first key local feature and the second key local features corresponding to each alternative image includes: Calculating the similarity values between the first key local feature and the second key local features according to the Term Frequency-Inverse Document Frequency (TF-IDF) algorithm to obtain local similarity values corresponding to each alternative image; wherein, the local similarity values correspond to the alternative images one by one; Determining the target image according to the multiple local similarity values, or determining the target image according to the multiple local similarity values and the multiple global similarity values.

7. The method according to claim 6, characterized in that, Determining the target image according to the multiple local similarity values includes: Determining the alternative image corresponding to the largest local similarity value as the target image.

8. The method according to claim 6, wherein Determining the target image according to the multiple local similarity values and the multiple global similarity values includes: Performing feature fusion on the multiple local similarity values and the multiple global similarity values to obtain multiple target similarity values, wherein the multiple target similarity values correspond to the multiple sample images one by one; Determining the sample image corresponding to the largest target similarity value as the target image.

9. The method according to any one of claims 1-5, 7-8, characterized in that, Before performing global feature extraction on the image to be retrieved and a preset plurality of sample images respectively, it further includes: Obtaining preset retrieval parameters, where the retrieval parameters are used to characterize the accuracy requirement for retrieving the image to be retrieved; Determining the preset sample images according to the retrieval parameters.

10. An image retrieval device, characterized in that, Includes: An acquisition module for acquiring the image to be retrieved; A global retrieval module for performing global feature extraction on the image to be retrieved and a preset plurality of sample images respectively through a convolutional neural network trained to convergence, and then performing feature dimensionality reduction and normalization processing to obtain a first global feature corresponding to the image to be retrieved and a second global feature corresponding to the sample images; wherein, different types of images to be retrieved correspond to different convolutional neural networks; the plurality of sample images are determined based on the retrieval parameters, and different retrieval parameters correspond to different retrieval ranges; the retrieval parameters are business identifiers; according to the first global feature and the plurality of second global features, obtaining a plurality of alternative images that meet the preset similarity parameters, wherein the number of the alternative images is less than the number of the sample images; A local retrieval module extracts local features from the image to be retrieved and the multiple alternative images respectively, obtaining a plurality of first local features corresponding to the image to be retrieved and a plurality of second local features corresponding to the alternative images; determines a first key local feature among the plurality of first local features according to preset weight parameters of each local feature, and determines, based on the bag-of-words model, a second key local feature that matches the first key local feature among the plurality of second local features corresponding to each alternative image; and determines a target image with the highest similarity to the image to be retrieved according to the similarity values between the first key local feature and the second key local features corresponding to each alternative image.

11. An electronic device, characterized in that, Comprising: a memory, a processor, and a computer program; wherein, the computer program is stored in the memory and is configured to be executed by the processor to implement the image retrieval method according to any one of claims 1-9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the image retrieval method according to any one of claims 1-9.

Images