Image retrieval model optimization method, image retrieval method, device and storage medium

Abstract

The present application relates to an image retrieval model optimization method, an image retrieval method, an apparatus and a storage medium, wherein the method obtains a first image feature of each sample image in a training data set to obtain an image feature set corresponding to the training data set, performs image feature extraction on any sample image in the training data set through an image retrieval model to be optimized to obtain a second image feature of the sample image, and performs image feature extraction on the sample image through an image feature extraction model to obtain a third image feature of the sample image, determines the loss of model optimization based on the second image feature, the third image feature of the sample image and the image feature set corresponding to the training data set, and obtains an optimized image retrieval model based on the loss, so that the image retrieval model to be optimized can truly learn the relationship information between the sample images, and completes the optimization of the image retrieval model to be optimized to obtain an image retrieval model with better performance.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to an image retrieval model optimization method, an image retrieval method, a device and a storage medium. Background Art

[0002] With the rise of deep learning technology, the development of image retrieval has been greatly promoted. Neural networks currently have irreplaceable advantages in extracting image features, and are therefore increasingly being used in image retrieval, a field that is extremely dependent on the quality of image features.

[0003] In related technologies, some knowledge in the field of metric learning is increasingly being used to improve image retrieval models. The intuitive understanding is that image samples of the same category should be closer in feature space, and image samples of different categories should be farther apart in feature space. However, current methods for improving image retrieval models are usually limited to the number of a single batch (samples), and the feature relationships of interest are limited to the current iteration of the network and the current batch. However, since image retrieval tasks usually have a large number of categories and a small number of image samples in each category, the image retrieval model improved by current methods is not global. Summary of the invention

[0004] Based on this, it is necessary to provide an image retrieval model optimization method, image retrieval method, device and storage medium that can be global in order to address the problem that the traditional method of improving the image retrieval model is not global.

[0005] A method for optimizing an image retrieval model, the method comprising:

[0006] Acquire a first image feature of each sample image in a training data set to obtain an image feature set corresponding to the training data set, wherein the training data set includes a plurality of sample images;

[0007] For any sample image in the training data set, extract image features through the image retrieval model to be optimized to obtain a second image feature of the sample image;

[0008] Extracting image features from the sample image using an image feature extraction model to obtain a third image feature of the sample image, wherein the image feature extraction model is obtained after momentum updating based on the image retrieval model to be optimized;

[0009] Determining a loss of model optimization according to the second image feature and the third image feature of the sample image and a set of image features corresponding to the training data set;

[0010] The image retrieval model to be optimized is back-propagated according to the loss to obtain an optimized image retrieval model.

[0011] In one of the embodiments, determining the loss of model optimization based on the second image feature, the third image feature of the sample image and the image feature set corresponding to the training data set includes: updating the first image feature in the image feature set corresponding to the sample image according to the third image feature of the sample image to obtain an updated image feature set; determining the loss of model optimization based on the updated image feature set and the second image feature of the sample image.

[0012] In one of the embodiments, determining the loss of model optimization based on the updated image feature set and the second image feature of the sample image includes: obtaining positive features and negative features corresponding to the sample image based on the updated image feature set; calculating the loss of model optimization based on the positive features, negative features corresponding to the sample image and the second image feature of the sample image.

[0013] In one of the embodiments, obtaining positive features and negative features corresponding to the sample image based on the updated image feature set includes: obtaining an updated first image feature corresponding to the sample image in the updated image feature set, and determining it as the positive feature corresponding to the sample image; obtaining other first image features in the updated image feature set except the positive feature corresponding to the sample image, and determining them as the negative feature corresponding to the sample image.

[0014] In one embodiment, the loss of model optimization is calculated according to the positive feature, negative feature and second image feature of the sample image, including: according to the positive feature, negative feature and second image feature of the sample image, the loss of model optimization is calculated using an information noise convergence estimation loss function, wherein the information noise convergence estimation loss function Wherein, q is the second image feature corresponding to the sample image, t + is the positive feature corresponding to the sample image, t i- is the i-th negative example feature corresponding to the sample image, K is the number of the negative example features, and τ is a hyperparameter.

[0015] In one of the embodiments, the step of acquiring the first image feature of each sample image in the training data set to obtain an image feature set corresponding to the training data set includes: using an initial image feature extraction model to perform image feature extraction on each sample image in the training data set to obtain the first image feature of each sample image; and generating an image feature set corresponding to the training data set based on the first image feature of each sample image in the training data set.

[0016] In one of the embodiments, for any sample image in the training data set, image feature extraction is performed through the image retrieval model to be optimized to obtain a second image feature of the sample image, including: for any sample image in the training data set, the sample image is input into the image retrieval model to be optimized to obtain a second original image feature corresponding to the sample image; and the second original image feature is subjected to affine transformation and norm normalization processing to obtain a corresponding processed second image feature.

[0017] In one embodiment, the image feature extraction of the sample image by an image feature extraction model to obtain a third image feature of the sample image includes: momentum updating the image feature extraction model based on the image retrieval model to be optimized to obtain an updated image feature extraction model; inputting the sample image into the momentum updated image feature extraction model to obtain a third original image feature corresponding to the sample image; and performing affine transformation and norm normalization processing on the third original image feature to obtain a corresponding processed third image feature.

[0018] An image retrieval method, the method comprising:

[0019] Get the image to be retrieved;

[0020] The image to be retrieved is input into the optimized image retrieval model obtained by the image retrieval model optimization method as described above to obtain the corresponding target image.

[0021] An image retrieval model optimization device, the device comprising:

[0022] An image feature set acquisition module, used to acquire a first image feature of each sample image in a training data set to obtain an image feature set corresponding to the training data set, wherein the training data set includes a plurality of sample images;

[0023] A second image feature extraction module is used to extract image features from any sample image in the training data set through the image retrieval model to be optimized to obtain a second image feature of the sample image;

[0024] A third image feature extraction module, configured to extract image features from the sample image through an image feature extraction model to obtain a third image feature of the sample image, wherein the image feature extraction model is obtained after momentum updating based on the image retrieval model to be optimized;

[0025] A loss acquisition module, used to determine the loss of model optimization according to the second image feature, the third image feature of the sample image and the image feature set corresponding to the training data set;

[0026] The model optimization module is used to perform back-propagation on the image retrieval model to be optimized according to the loss to obtain an optimized image retrieval model.

[0027] An image retrieval device, comprising:

[0028] A module for acquiring images to be retrieved, used for acquiring images to be retrieved;

[0029] The retrieval processing module is used to input the image to be retrieved into the optimized image retrieval model obtained by the image retrieval model optimization method as described above, to obtain the corresponding target image.

[0030] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0031] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0032] The above-mentioned image retrieval model optimization method, image retrieval method, device and storage medium obtain the first image feature of each sample image in the training data set to obtain the image feature set corresponding to the training data set. For any sample image in the training data set, image feature extraction is performed through the image retrieval model to be optimized to obtain the second image feature of the sample image, and image feature extraction is performed on the sample image through the image feature extraction model to obtain the third image feature of the sample image. According to the second image feature, the third image feature of the sample image and the image feature set corresponding to the training data set, the loss of model optimization is determined, and the image retrieval model to be optimized is back-propagated according to the loss to obtain the optimized image retrieval model, so that the image retrieval model to be optimized can truly learn the relationship information between sample images, and complete the optimization of the image retrieval model to be optimized to obtain an image retrieval model with better performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1A schematic diagram of a flow chart of an image retrieval model optimization method in one embodiment;

[0034] Figure 2 A schematic diagram of a flow chart of the steps of obtaining an image feature set in one embodiment;

[0035] Figure 3 A schematic diagram of a flow chart of a step of obtaining a second image feature in an embodiment;

[0036] Figure 4 It is a schematic diagram of a flow chart of a step of obtaining a third image feature in one embodiment;

[0037] Figure 5 A schematic diagram of a process for determining a loss step for model optimization in one embodiment;

[0038] Figure 6 A schematic diagram of a flow chart of a loss step of calculating model optimization in one embodiment;

[0039] Figure 7 A schematic diagram of a flow chart of an image retrieval model optimization method in another embodiment;

[0040] Figure 8 is a schematic diagram of a flow chart of an image retrieval method in one embodiment;

[0041] Fig. 9 is a structural block diagram of an image retrieval model optimization device in one embodiment;

[0042] Fig.10 is a structural block diagram of an image retrieval device in one embodiment;

[0043] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] In one embodiment, a method for optimizing an image retrieval model is provided. Figure 1 As shown, the following steps are included:

[0046] Step 110, obtaining the first image feature of each sample image in the training data set, and obtaining an image feature set corresponding to the training data set.

[0047] Among them, the training data set includes multiple sample images, and the training data set is sample data used to optimize the image retrieval model to be optimized so that it can improve the retrieval effect. The image retrieval model to be optimized can be any neural network model used for image retrieval in traditional technology. The first image feature is the image feature result obtained after image feature extraction of the sample image in the training data set. In this embodiment, when it is necessary to optimize a certain image retrieval model, it is first necessary to obtain a training data set for model optimization, and then extract image features for each sample image in the training data set, so as to obtain the first image features corresponding to each sample image, and based on the first image features corresponding to each sample image in the training data set, an image feature set corresponding to the training data set is obtained, that is, the image feature set includes the first image feature of each sample image.

[0048] Step 120: For any sample image in the training data set, image feature extraction is performed using the image retrieval model to be optimized to obtain a second image feature of the sample image.

[0049] Among them, the image retrieval model to be optimized can be any neural network model used for image retrieval in traditional technology, for example, it can be a neural network model used for image search in search engines (Google, Baidu), a neural network model used for similar product search in e-commerce websites (Taobao, Amazon, eBay), a neural network model used for similar content recommendation in social platforms (Pinterest), etc. The second image feature is the image feature result obtained after the sample image passes through the image retrieval model to be optimized. Specifically, in this embodiment, for any sample image in the training data set, the sample image is input into the image retrieval model to be optimized, so as to obtain the second image feature corresponding to the sample image.

[0050] Step 130: extract image features from the sample image using an image feature extraction model to obtain a third image feature of the sample image.

[0051] The image feature extraction model is obtained after momentum updating based on the image retrieval model to be optimized. The third image feature is the image feature result obtained after the sample image passes through the image feature extraction model. Specifically, for any sample image in the training data set (i.e., the sample image in the above step 120), the sample image is input into the image feature extraction model obtained after momentum updating based on the image retrieval model to be optimized, so as to obtain the third image feature corresponding to the sample image.

[0052] Step 140, determining the loss of model optimization according to the second image feature and the third image feature of the sample image and the image feature set corresponding to the training data set.

[0053] Among them, the loss of model optimization represents the relationship information between sample images, which is used to optimize the image retrieval model to be optimized. In this embodiment, the loss of model optimization is determined by the second image feature and the third image feature corresponding to any sample image in the training data set and the image feature set corresponding to the training data set, and then the model parameters of the image retrieval model to be optimized are adjusted based on the loss of model optimization, so that the image retrieval model to be optimized can truly learn the relationship information between sample images, so as to complete the optimization of the image retrieval model to be optimized and improve the retrieval effect of the optimized image retrieval model.

[0054] Step 150, back-propagating the image retrieval model to be optimized according to the loss to obtain an optimized image retrieval model.

[0055] Specifically, after the loss of model optimization is obtained through the above steps, the image retrieval model to be optimized can be back-propagated according to the loss to complete the calculation of an iterative training of the image retrieval model to be optimized, thereby updating the model parameters of the image retrieval model to be optimized. It can be understood that for each sample image in the training data set, the method of steps 120 to 150 can be used to process, so that the image retrieval model to be optimized can truly learn the relationship information between the sample images, so as to complete the optimization of the image retrieval model to be optimized and improve the retrieval effect of the optimized image retrieval model.

[0056] The above-mentioned image retrieval model optimization method obtains the first image feature of each sample image in the training data set to obtain the image feature set corresponding to the training data set. For any sample image in the training data set, image feature extraction is performed through the image retrieval model to be optimized to obtain the second image feature of the sample image, and image feature extraction is performed on the sample image through the image feature extraction model to obtain the third image feature of the sample image. According to the second image feature, the third image feature of the sample image and the image feature set corresponding to the training data set, the loss of model optimization is determined, and the image retrieval model to be optimized is back-propagated according to the loss to obtain the optimized image retrieval model, so that the image retrieval model to be optimized can truly learn the relationship information between sample images, and complete the optimization of the image retrieval model to be optimized to obtain an image retrieval model with better performance.

[0057] In one embodiment, Figure 2As shown, in step 110, the first image feature of each sample image in the training data set is obtained to obtain an image feature set corresponding to the training data set, which may specifically include the following steps:

[0058] Step 111 , using an initial image feature extraction model to extract image features from each sample image in the training data set, to obtain a first image feature of each sample image.

[0059] The initial image feature extraction model is obtained based on the image retrieval model to be optimized. In this embodiment, the structure of the image retrieval model to be optimized is given, and it is initialized based on the pre-training parameters of ImageNet (a large visual database for visual object recognition software research), so as to obtain the image retrieval model to be optimized, and the initial image feature extraction model is exactly the same as the image retrieval model to be optimized. In this embodiment, by inputting each sample image in the training data set into the initial image feature extraction model, the first image feature corresponding to each sample image is obtained.

[0060] Step 112: Generate an image feature set corresponding to the training data set based on the first image feature of each sample image in the training data set.

[0061] Specifically, for each sample image in the training data set, image feature extraction is performed to obtain the first image feature corresponding to each sample image, and based on the one-to-one corresponding first image feature of each sample image in the training data set, an image feature set corresponding to the training data set is obtained.

[0062] In the above embodiment, an initial image feature extraction model is used to extract image features from each sample image in the training data set to obtain the first image feature of each sample image, and based on the first image feature of each sample image in the training data set, an image feature set corresponding to the training data set is generated. Since the image feature set stores image features of multiple sample images, the problem of insufficient number of samples in a single batch is solved.

[0063] In one embodiment, Figure 3 As shown, in step 120, for any sample image in the training data set, image feature extraction is performed through the image retrieval model to be optimized to obtain a second image feature of the sample image, which may specifically include the following steps:

[0064] Step 121: for any sample image in the training data set, the sample image is input into the image retrieval model to be optimized to obtain a second original image feature corresponding to the sample image.

[0065] The image retrieval model to be optimized usually includes a backbone network and a fully connected layer plus norm normalization. The image retrieval model to be optimized can be updated according to normal gradients. The second original image feature is the output of the sample image after passing through the backbone network of the image retrieval model to be optimized, that is, the original image feature obtained after the sample image passes through the backbone network of the image retrieval model to be optimized but has not yet been processed by the fully connected layer and norm normalization. Specifically, for any sample image in the training data set, the second original image feature corresponding to the sample image is obtained by inputting the sample image into the image retrieval model to be optimized.

[0066] Step 122, performing affine transformation and norm normalization processing on the second original image feature to obtain a corresponding processed second image feature.

[0067] Among them, affine transformation, also known as affine mapping, refers to the process of transforming a vector space into another vector space by performing a linear transformation and a translation in geometry. Norm normalization can be processed by L2 norm normalization. L2 norm normalization is to divide each dimensional data x1, x2, ..., xn of vector X by ||x||2 (i.e., L2 norm) to obtain a new vector. L2 norm normalization can eliminate the influence of units between features and improve the convergence speed of the learning model. In this embodiment, by performing an affine transformation on the second original image feature, the dimension of the second original image feature output by the backbone network of the image retrieval model to be optimized is consistent with the dimension of the first original image feature output by the initial image feature extraction model. Then, the second original image feature after affine transformation is subjected to norm normalization processing to obtain the corresponding processed second image feature.

[0068] In the above embodiment, for any sample image in the training data set, the sample image is input into the image retrieval model to be optimized to obtain a second original image feature corresponding to the sample image, and the second original image feature is affine transformed and norm normalized to obtain the corresponding processed second image feature, thereby facilitating the image retrieval model to be optimized to learn the relationship information between sample images.

[0069] In one embodiment, Figure 4 As shown, in step 130, image features of the sample image are extracted by using an image feature extraction model to obtain a third image feature of the sample image, which may specifically include the following steps:

[0070] Step 131 , performing momentum update on the image feature extraction model based on the image retrieval model to be optimized to obtain an updated image feature extraction model.

[0071] Among them, the initial image feature extraction model is exactly the same as the initial image retrieval model to be optimized, which also includes a backbone network and a fully connected layer plus norm normalization. Specifically, the image feature extraction model and the initial image retrieval model to be optimized use the same network structure and initialization parameters. The image retrieval model to be optimized can be updated according to the normal gradient, while the image feature extraction model can be updated by the sliding average of the iteration number of the image retrieval model to be optimized (i.e., momentum update). Specifically, if the model parameter of the image retrieval model to be optimized is θ q , the model parameters of the image feature extraction model before momentum update are θ k , then θ k+1 =mθ k +(1-m)θ q , where θ k+1 is the model parameter of the image feature extraction model after momentum update, m is the momentum size, which may be a hyperparameter preset before network training. In this embodiment, the image feature extraction model is momentum updated based on the image retrieval model to be optimized, thereby obtaining an updated image feature extraction model.

[0072] Step 132: input the sample image into the momentum updated image feature extraction model to obtain a third original image feature corresponding to the sample image.

[0073] Similarly, the third original image feature is the output of the sample image after the backbone network of the image feature extraction model after momentum update, that is, the original image feature obtained after the sample image passes through the backbone network of the image feature extraction model after momentum update but before the full connection layer and norm normalization processing. Specifically, by inputting the sample image (that is, the sample image determined in the above step 121) into the image feature extraction model after momentum update, the third original image feature corresponding to the sample image is obtained.

[0074] Step 133, performing affine transformation and norm normalization processing on the third original image feature to obtain the corresponding processed third image feature.

[0075] Specifically, the third original image feature is subjected to affine transformation and norm normalization processing to obtain the corresponding processed third image feature. The affine transformation and norm normalization processing can be performed in the same manner as in step 122, which will not be described in detail in this embodiment.

[0076] In the above embodiment, the image feature extraction model is momentum updated based on the image retrieval model to be optimized to obtain an updated image feature extraction model, and the third original image feature corresponding to the sample image is obtained by inputting the sample image into the image feature extraction model after momentum update, and the third original image feature is affine transformed and norm normalized to obtain the corresponding processed third image feature having the same dimension as the first image feature, thereby facilitating the image retrieval model to be optimized to learn the relationship features between sample images.

[0077] In one embodiment, Figure 5 As shown, in step 140, the loss of model optimization is determined according to the second image feature, the third image feature of the sample image and the image feature set corresponding to the training data set, which may specifically include the following steps:

[0078] Step 141: Update the first image feature corresponding to the sample image in the image feature set according to the third image feature of the sample image to obtain an updated image feature set.

[0079] Since the image feature set includes the first image feature of each sample image, and the first image feature is obtained by extracting the image features of each sample image in the training data set using the initial image feature extraction model. The third image feature is obtained by extracting the image features of a sample image in the training data set using the image feature extraction model after momentum update. Therefore, in this embodiment, based on the third image feature of a sample image, the first image feature corresponding to the sample image in the image feature set is updated, that is, the first image feature corresponding to the sample image in the image feature set is updated to the third image feature of the sample image, thereby obtaining the updated image feature set, thereby ensuring the consistency and slow variation of the features in the image feature set.

[0080] Step 142, determining the loss of model optimization according to the updated image feature set and the second image feature of the sample image.

[0081] Specifically, the loss of model optimization is determined according to the updated image feature set and the second image feature of the sample image, and then the model parameters of the image retrieval model to be optimized are adjusted based on the loss of model optimization, so that the image retrieval model to be optimized can truly learn the relationship information between sample images to complete the optimization of the image retrieval model to be optimized.

[0082] In one embodiment, Figure 6 As shown, in step 142, the loss of model optimization is determined according to the updated image feature set and the second image feature of the sample image, which may specifically include the following steps:

[0083] Step 601: Based on the updated image feature set, obtain positive feature and negative feature corresponding to the sample image.

[0084] Among them, the positive feature refers to the first image feature corresponding to any sample image in the training data set in the updated image feature set, and the negative feature refers to the other first image features in the updated image feature set except the positive feature corresponding to the sample image. That is, for any sample image in the training data set, the corresponding positive feature is the first image feature corresponding to the sample image in the updated image feature set, and the negative feature is all the first image features in the updated image feature set except the positive feature corresponding to the sample image. Based on this, the positive feature and negative feature corresponding to any sample image in the training data set can be determined.

[0085] Step 602, calculating the loss of model optimization according to the positive example features, negative example features corresponding to the sample image and the second image features of the sample image.

[0086] Specifically, this embodiment can use the information noise convergence estimation loss function (info Noise-Constrastive Estimation Loss) to calculate the loss of model optimization, where the information noise convergence estimation loss function Wherein, q is the second image feature corresponding to the sample image (i.e., the sample image determined in step 121 above), t + is the positive feature corresponding to the sample image, t i- is the i-th negative feature corresponding to the sample image, K is the number of negative features, and τ is a configurable hyperparameter.

[0087] like Figure 7 As shown, the following is based on Figure 7 The image retrieval model optimization method in this application is further explained. In this case, the backbone network (i.e., the backbone network) takes GoogleNet as an example, and the pre-trained parameters on ImageNet are given for initialization. The specific steps are as follows:

[0088] 1. If Figure 7As shown, model_q (i.e., 701 in the figure) is the image retrieval model to be optimized, which includes backbone_q (googlenet) and head_q (single-layer fully connected layer plus L2 norm normalization). Model_k is an image feature extraction model, which can also be called a momentum encoder. It is a network exactly the same as model_q, including backbone_k and head_k. Initially, model_k uses the same initialization parameters as model_q. Model_q is updated according to the normal gradient, while model_k is directly updated using the sliding average of the model_q parameters for each iteration. The specific update method is shown in the following formula:

[0089] θ k+1 =mθ k +(1-m)θ q , where θ k+1 is the model parameter of model_k after momentum update, θ k is the model parameter of model_k before momentum update, θ q is the model parameter of model_q, m is the momentum size, which can be a hyperparameter pre-set before network training.

[0090] 2. Obtain a training data set for model optimization. Assume that it contains N sample images. After inputting each sample image into the initial model_k, obtain the corresponding first image feature, i.e., f1, f2, …, f N , based on the N image features, the corresponding image feature set f is obtained i , and use it as a Memory Bank with a size of N×D, where N is the number of elements in the set and D is the feature dimension.

[0091] 3. For model_q, for any sample image m in the training data set, it outputs the corresponding feature (i.e., the second original image feature) through backbone_q, denoted as f m , then f m ∈R d , d is the feature dimension.

[0092] In this embodiment, by m Perform an affine transformation so that f m With f i For the same feature dimension, in this embodiment, the specific affine transformation method is as follows:

[0093] q m ′=Wfm +b; where W∈R d×D , W represents the parameter matrix of affine transformation, b is the bias, and d is f m The feature dimension of f m The target feature dimension for affine transformation, f m is the second original image feature obtained after the sample image m is output by model_q, q m ′ is the pair f m The corresponding image features obtained after affine transformation.

[0094] Then, q m ' Perform L2 norm normalization processing (affine transformation and L2 norm normalization processing are completed by head_q in the figure), so as to obtain the corresponding processed second image feature, i.e., q. The L2 norm normalization processing process is as follows:

[0095] Among them, q m ′ is the pair f m The corresponding image features obtained after affine transformation, ||q m ′||2 is the image feature q m ′ corresponds to the L2 norm, D is the feature dimension, q m ' j Q m ′ is the j-th dimension image feature.

[0096] Among them, q is the m ′ is the second image feature obtained after L2 norm normalization.

[0097] 4. Similarly, for the model_k network, the same sample image m outputs feature k (i.e., the third original image feature) through backbone_k and head_k. If it is denoted as k, the corresponding feature stored in its memory bank is updated according to the index of the sample image m, and the updated memory bank is obtained.

[0098] 5. For sample image m, it corresponds to the first image feature t in Memory Bank i+ is the positive example feature (for the convenience of explanation, t is used in the following formula + ), while other first image features in Memory Bank is the negative feature corresponding to the sample image m (for the convenience of explanation, t is used in the following formula i- , where K is the number of negative features.

[0099] 6. For the sample image m, the infoNCE loss corresponding to the sample image is calculated based on the positive and negative features in the corresponding Memory Bank obtained above:

[0100] Among them, q is the second image feature corresponding to the sample image m, t + is the positive feature corresponding to the sample image m, t i- is the i-th negative feature corresponding to the sample image m, K is the number of negative features, and τ is a configurable hyperparameter.

[0101] Based on this, for any sample image n, the infoNCE loss is calculated based on the positive and negative features in the Memory Bank:

[0102] Among them, q n is the second image feature corresponding to any sample image n, t n+ is the positive feature corresponding to the sample image n, is the jth negative feature corresponding to sample image n, K is the number of negative features, and τ is a configurable hyperparameter.

[0103] 7. Loss is returned to update the network parameters of model_q according to the gradient.

[0104] The image retrieval model optimization method provided in this embodiment introduces a memory bank to store feature vectors of a large number of sample images, thereby solving the problem of insufficient number of sample images in a single batch, and uses the infoNCE loss that performs well in representation learning tasks to extract relationship information between samples. The Memory Bank is used to ensure feature reuse, and a large number of negative samples are provided. The momentum encoder is used to ensure the consistency and slow variation of features in the Memory Bank, thereby solving the defect that the features in the original Memory Bank come from different iterations and different samples, thereby greatly improving the effect of optimizing the image retrieval model.

[0105] In one embodiment, Figure 8 As shown, an image retrieval method is provided, comprising the following steps:

[0106] Step 810, obtaining the image to be retrieved.

[0107] Among them, the image to be retrieved is the image to be retrieved. Specifically, the image retrieval includes but is not limited to image retrieval in a search engine, image retrieval in an e-commerce website to search for similar products, image retrieval in a social platform to recommend similar content, etc.

[0108] Step 820: input the image to be retrieved into the image retrieval model to obtain the corresponding target image.

[0109] Among them, the image retrieval model is as above Figures 1 to 7 The optimized image retrieval model obtained by the image retrieval model optimization method is understood to be Figures 1 to 7 After the image retrieval model optimization method obtains the image retrieval model, the image retrieval model can also be trained based on the scenarios involving image retrieval in actual applications, so that it can learn the features in the specific scenarios, and use the trained image retrieval model to perform image retrieval on the image to be retrieved, thereby obtaining a target image whose similarity with the image to be retrieved reaches a target value, wherein the target value can be set according to the actual application scenario.

[0110] Since the image retrieval model used for image retrieval is Figures 1 to 7 The image retrieval model is obtained by the image retrieval model optimization method. Therefore, the image retrieval model can truly learn the relationship features between sample images, thereby achieving the effect of optimizing the image retrieval model. When it is used for specific image retrieval tasks after training, it can also improve the effect of specific image retrieval tasks.

[0111] It should be understood that although Figure 1-8 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1-8 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0112] In one embodiment, Fig. 9 As shown, an image retrieval model optimization device is provided, including: an image feature set acquisition module 901, a second image feature extraction module 902, a third image feature extraction module 903, a loss acquisition module 904 and a model optimization module 905, wherein:

[0113] An image feature set acquisition module 901 is used to acquire a first image feature of each sample image in a training data set to obtain an image feature set corresponding to the training data set, wherein the training data set includes a plurality of sample images;

[0114] A second image feature extraction module 902 is used to extract image features from any sample image in the training data set by using the image retrieval model to be optimized to obtain a second image feature of the sample image;

[0115] A third image feature extraction module 903, configured to extract image features from the sample image using an image feature extraction model to obtain a third image feature of the sample image, wherein the image feature extraction model is obtained after momentum updating based on the image retrieval model to be optimized;

[0116] A loss acquisition module 904, configured to determine a loss of model optimization according to the second image feature and the third image feature of the sample image and the image feature set corresponding to the training data set;

[0117] The model optimization module 905 is used to perform back-propagation on the image retrieval model to be optimized according to the loss to obtain an optimized image retrieval model.

[0118] In one embodiment, the loss acquisition module includes: an updating unit, used to update the first image feature corresponding to the sample image in the image feature set according to the third image feature of the sample image, so as to obtain an updated image feature set; and a loss determination unit, used to determine the loss of model optimization according to the updated image feature set and the second image feature of the sample image.

[0119] In one embodiment, the loss determination unit includes: a feature determination subunit, used to obtain positive example features and negative example features corresponding to the sample image based on an updated image feature set; a loss calculation subunit, used to calculate the loss of model optimization based on the positive example features, negative example features corresponding to the sample image and the second image features of the sample image.

[0120] In one embodiment, the feature determination subunit is specifically used to: obtain the updated first image feature corresponding to the sample image in the updated image feature set, and determine it as the positive example feature corresponding to the sample image; obtain other first image features in the updated image feature set except the positive example feature corresponding to the sample image, and determine them as the negative example features corresponding to the sample image.

[0121] In one embodiment, the loss calculation subunit is specifically used to calculate the loss of the model optimization using an information noise convergence estimation loss function according to the positive example features, negative example features and second image features corresponding to the sample image, wherein the information noise convergence estimation loss function Wherein, q is the second image feature corresponding to the sample image, t +is the positive feature corresponding to the sample image, t i- is the i-th negative example feature corresponding to the sample image, K is the number of the negative example features, and τ is a hyperparameter.

[0122] In one embodiment, the image feature set acquisition module is specifically used to: use an initial image feature extraction model to extract image features from each sample image in the training data set to obtain a first image feature of each sample image; and based on the first image feature of each sample image in the training data set, generate an image feature set corresponding to the training data set.

[0123] In one of the embodiments, the second image feature extraction module is specifically used to: for any sample image in the training data set, input the sample image into the image retrieval model to be optimized to obtain a second original image feature corresponding to the sample image; perform affine transformation and norm normalization processing on the second original image feature to obtain the corresponding processed second image feature.

[0124] In one of the embodiments, the third image feature extraction module is specifically used to: perform momentum update on the image feature extraction model based on the image retrieval model to be optimized to obtain an updated image feature extraction model; input the sample image into the momentum updated image feature extraction model to obtain a third original image feature corresponding to the sample image; perform affine transformation and norm normalization on the third original image feature to obtain a corresponding processed third image feature.

[0125] For the specific definition of the image retrieval model optimization device, please refer to the definition of the image retrieval model optimization method above, which will not be repeated here. The various modules in the above-mentioned image retrieval model optimization device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0126] In one embodiment, Fig.10 As shown, an image retrieval device is provided, including a to-be-retrieved image acquisition module 1001 and a retrieval processing module 1002, wherein:

[0127] The to-be-searched image acquisition module 1001 is used to acquire the to-be-searched image;

[0128] The retrieval processing module 1002 is used to input the image to be retrieved into the optimized image retrieval model obtained by the image retrieval model optimization method as described above, to obtain the corresponding target image.

[0129] For the specific definition of the image retrieval device, please refer to the definition of the image retrieval method above, which will not be repeated here. Each module in the above-mentioned image retrieval device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0130] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.11 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the image feature set corresponding to the training data set. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image retrieval model optimization method or an image retrieval method is implemented.

[0131] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0132] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0133] Acquire a first image feature of each sample image in a training data set to obtain an image feature set corresponding to the training data set, wherein the training data set includes a plurality of sample images;

[0134] For any sample image in the training data set, extract image features through the image retrieval model to be optimized to obtain a second image feature of the sample image;

[0135] Extracting image features from the sample image using an image feature extraction model to obtain a third image feature of the sample image, wherein the image feature extraction model is obtained after momentum updating based on the image retrieval model to be optimized;

[0136] Determining a loss of model optimization according to the second image feature and the third image feature of the sample image and a set of image features corresponding to the training data set;

[0137] The image retrieval model to be optimized is back-propagated according to the loss to obtain an optimized image retrieval model.

[0138] In one embodiment, when the processor executes the computer program, the following steps are also implemented: based on the third image feature of the sample image, the first image feature corresponding to the sample image in the image feature set is updated to obtain an updated image feature set; based on the updated image feature set and the second image feature of the sample image, the loss of model optimization is determined.

[0139] In one embodiment, when the processor executes the computer program, the following steps are also implemented: based on the updated image feature set, the positive example features and negative example features corresponding to the sample image are obtained; according to the positive example features and negative example features corresponding to the sample image and the second image features of the sample image, the loss of model optimization is calculated.

[0140] In one embodiment, when the processor executes the computer program, the following steps are also implemented: obtaining an updated first image feature corresponding to the sample image in the updated image feature set, and determining it as a positive feature corresponding to the sample image; obtaining other first image features in the updated image feature set except the positive feature corresponding to the sample image, and determining them as negative features corresponding to the sample image.

[0141] In one embodiment, when the processor executes the computer program, the following steps are further implemented: according to the positive example features, the negative example features and the second image features corresponding to the sample image, the loss of the model optimization is calculated using the information noise convergence estimation loss function, wherein the information noise convergence estimation loss function Wherein, q is the second image feature corresponding to the sample image, t + is the positive feature corresponding to the sample image, t i- is the i-th negative example feature corresponding to the sample image, K is the number of the negative example features, and τ is a hyperparameter.

[0142] In one embodiment, when the processor executes the computer program, the following steps are also implemented: using an initial image feature extraction model to extract image features from each sample image in the training data set to obtain a first image feature of each sample image; based on the first image feature of each sample image in the training data set, generating an image feature set corresponding to the training data set.

[0143] In one embodiment, when the processor executes the computer program, the following steps are also implemented: for any sample image in the training data set, the sample image is input into the image retrieval model to be optimized to obtain a second original image feature corresponding to the sample image; and the second original image feature is affine transformed and norm normalized to obtain a corresponding processed second image feature.

[0144] In one embodiment, when the processor executes the computer program, the following steps are also implemented: momentum updating the image feature extraction model based on the image retrieval model to be optimized to obtain an updated image feature extraction model; inputting the sample image into the momentum updated image feature extraction model to obtain a third original image feature corresponding to the sample image; performing affine transformation and norm normalization processing on the third original image feature to obtain a corresponding processed third image feature.

[0145] In one embodiment, when the processor executes the computer program, the following steps are also implemented:

[0146] Get the image to be retrieved;

[0147] The image to be retrieved is input into the optimized image retrieval model obtained by the image retrieval model optimization method as described above to obtain the corresponding target image.

[0148] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0149] Acquire a first image feature of each sample image in a training data set to obtain an image feature set corresponding to the training data set, wherein the training data set includes a plurality of sample images;

[0150] For any sample image in the training data set, extract image features through the image retrieval model to be optimized to obtain a second image feature of the sample image;

[0151] Extracting image features from the sample image using an image feature extraction model to obtain a third image feature of the sample image, wherein the image feature extraction model is obtained after momentum updating based on the image retrieval model to be optimized;

[0152] Determining a loss of model optimization according to the second image feature and the third image feature of the sample image and a set of image features corresponding to the training data set;

[0153] The image retrieval model to be optimized is back-propagated according to the loss to obtain an optimized image retrieval model.

[0154] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: based on the third image feature of the sample image, the first image feature in the image feature set corresponding to the sample image is updated to obtain an updated image feature set; based on the updated image feature set and the second image feature of the sample image, the loss of model optimization is determined.

[0155] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: based on the updated image feature set, the positive features and negative features corresponding to the sample image are obtained; according to the positive features and negative features corresponding to the sample image and the second image features of the sample image, the loss of model optimization is calculated.

[0156] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining an updated first image feature corresponding to the sample image in the updated image feature set, and determining it as a positive example feature corresponding to the sample image; obtaining other first image features in the updated image feature set except the positive example feature corresponding to the sample image, and determining them as negative example features corresponding to the sample image.

[0157] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: according to the positive example features, the negative example features and the second image features corresponding to the sample image, the loss of the model optimization is calculated using the information noise convergence estimation loss function, wherein the information noise convergence estimation loss function Wherein, q is the second image feature corresponding to the sample image, t + is the positive feature corresponding to the sample image, t i- is the i-th negative example feature corresponding to the sample image, K is the number of the negative example features, and τ is a hyperparameter.

[0158] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: using an initial image feature extraction model to extract image features from each sample image in the training data set to obtain a first image feature of each sample image; based on the first image feature of each sample image in the training data set, generating an image feature set corresponding to the training data set.

[0159] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: for any sample image in the training data set, the sample image is input into the image retrieval model to be optimized to obtain a second original image feature corresponding to the sample image; and the second original image feature is affine transformed and norm normalized to obtain a corresponding processed second image feature.

[0160] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: momentum updating the image feature extraction model based on the image retrieval model to be optimized to obtain an updated image feature extraction model; inputting the sample image into the momentum updated image feature extraction model to obtain a third original image feature corresponding to the sample image; performing affine transformation and norm normalization processing on the third original image feature to obtain a corresponding processed third image feature.

[0161] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0162] Get the image to be retrieved;

[0163] The image to be retrieved is input into the optimized image retrieval model obtained by the image retrieval model optimization method as described above to obtain the corresponding target image.

[0164] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0165] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0166] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for optimizing an image retrieval model, characterized in that: The method comprises: Acquire a first image feature of each sample image in a training data set to obtain an image feature set corresponding to the training data set, wherein the training data set includes a plurality of sample images; For any sample image in the training data set, extract image features through the image retrieval model to be optimized to obtain a second image feature of the sample image; Extracting image features from the sample image using an image feature extraction model to obtain a third image feature of the sample image, wherein the image feature extraction model is obtained after momentum updating based on the image retrieval model to be optimized; Determining a loss of model optimization according to the second image feature and the third image feature of the sample image and a set of image features corresponding to the training data set; Performing back propagation on the image retrieval model to be optimized according to the loss to obtain an optimized image retrieval model; The determining the loss of model optimization according to the second image feature, the third image feature of the sample image and the image feature set corresponding to the training data set includes: updating the first image feature corresponding to the sample image in the image feature set according to the third image feature of the sample image to obtain an updated image feature set; The loss of model optimization is determined according to the updated image feature set and the second image feature of the sample image.

2. The method according to claim 1, characterized in that The step of determining the loss of model optimization according to the updated image feature set and the second image feature of the sample image comprises: Based on the updated image feature set, obtaining positive feature and negative feature corresponding to the sample image; The loss of model optimization is calculated according to the positive example features and negative example features corresponding to the sample image and the second image features of the sample image.

3. The method according to claim 2, characterized in that The acquiring, based on the updated image feature set, positive feature and negative feature corresponding to the sample image comprises: Obtaining an updated first image feature corresponding to the sample image in the updated image feature set, and determining it as a positive example feature corresponding to the sample image; The other first image features except the positive example features corresponding to the sample image in the updated image feature set are obtained and determined as negative example features corresponding to the sample image.

4. The method according to claim 3, characterized in that The calculating the loss of model optimization according to the positive example features and the negative example features corresponding to the sample image and the second image features of the sample image includes: According to the positive example features, negative example features and second image features corresponding to the sample image, the loss of the model optimization is calculated using an information noise convergence estimation loss function, wherein the information noise convergence estimation loss function Wherein, q is the second image feature corresponding to the sample image, t + is the positive feature corresponding to the sample image, t i- is the i-th negative example feature corresponding to the sample image, K is the number of the negative example features, and τ is a hyperparameter.

5. The method according to any one of claims 1 to 4, characterized in that: The step of acquiring the first image feature of each sample image in the training data set to obtain an image feature set corresponding to the training data set includes: Using an initial image feature extraction model to extract image features from each sample image in the training data set to obtain a first image feature of each sample image; Based on the first image feature of each sample image in the training data set, an image feature set corresponding to the training data set is generated.

6. The method according to any one of claims 1 to 4, characterized in that: For any sample image in the training data set, extracting image features through the image retrieval model to be optimized to obtain a second image feature of the sample image includes: For any sample image in the training data set, input the sample image into the image retrieval model to be optimized to obtain a second original image feature corresponding to the sample image; Affine transformation and norm normalization processing are performed on the second original image feature to obtain a corresponding processed second image feature.

7. The method according to any one of claims 1 to 4, characterized in that: The extracting the image features of the sample image by using the image feature extraction model to obtain the third image feature of the sample image includes: Performing momentum update on the image feature extraction model based on the image retrieval model to be optimized to obtain an updated image feature extraction model; Inputting the sample image into the momentum updated image feature extraction model to obtain a third original image feature corresponding to the sample image; Affine transformation and norm normalization processing are performed on the third original image feature to obtain a corresponding processed third image feature.

8. An image retrieval method, characterized in that: The method comprises: Get the image to be retrieved; The image to be retrieved is input into an optimized image retrieval model obtained by the image retrieval model optimization method according to any one of claims 1 to 7 to obtain a corresponding target image.

9. An image retrieval model optimization device, characterized in that: The device comprises: An image feature set acquisition module, used to acquire a first image feature of each sample image in a training data set to obtain an image feature set corresponding to the training data set, wherein the training data set includes a plurality of sample images; A second image feature extraction module is used to extract image features from any sample image in the training data set through the image retrieval model to be optimized to obtain a second image feature of the sample image; A third image feature extraction module, configured to extract image features from the sample image through an image feature extraction model to obtain a third image feature of the sample image, wherein the image feature extraction model is obtained after momentum updating based on the image retrieval model to be optimized; A loss acquisition module, used to determine the loss of model optimization according to the second image feature, the third image feature of the sample image and the image feature set corresponding to the training data set; A model optimization module, used for performing back propagation on the image retrieval model to be optimized according to the loss to obtain an optimized image retrieval model; The loss acquisition module is also used to: update the first image feature corresponding to the sample image in the image feature set according to the third image feature of the sample image to obtain an updated image feature set; determine the loss of model optimization according to the updated image feature set and the second image feature of the sample image.

10. An image retrieval device, characterized in that: The device comprises: A module for acquiring images to be retrieved, used for acquiring images to be retrieved; The retrieval processing module is used to input the image to be retrieved into the optimized image retrieval model obtained by the image retrieval model optimization method according to any one of claims 1 to 7 to obtain the corresponding target image.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images