A deep image retrieval method for small sample scenes

Abstract

The present invention discloses a deep image retrieval method for small sample scenarios, especially when the available learning samples are relatively sparse, the present invention is used to obtain more semantic information from a small number of samples by combining contrastive learning, so as to improve the representation and encoding capabilities of the deep hash model. First, training samples are sampled from a database, and positive and negative sample copies are generated by random enhancement. Then, a hash deep network is used to generate continuous features to retain more semantic information, and finally a binary code is generated by mapping. At the same time, the binary code of the database sample is maintained using the similarity matrix between the training samples and between the training samples and the database. After the model converges at the end of the training phase, for external samples, a deep hash network can be used to calculate its binary code, and then the binary code of the database can be directly retrieved. Compared with the prior art, the present invention can improve the efficiency and accuracy of image retrieval, and is more robust when facing small sample scenarios.

Description

Technical Field

[0001] The present invention relates to a deep image retrieval method for small sample scenes, belongs to the fields of computer vision and image retrieval, and is particularly suitable for small sample scenes with a small number of available training samples. Background Art

[0002] With the explosive growth of data, approximate nearest neighbor search has attracted more and more attention in the field of information retrieval. As a classic technique in approximate nearest neighbor search, the hash algorithm uses a hash function to encode high-dimensional data into binary codes while retaining the similarity information of the original data. Based on the short binary code representation, the similarity between the query sample and the sample in the database can be measured by the Hamming distance. Therefore, the approximate nearest neighbor search combined with the hash operation can achieve linear or sublinear time complexity, with significant advantages such as fast query speed and low storage cost.

[0003] Traditional hashing methods use manual methods (such as SIFT operators) to generate features for binary code learning. As deep learning has achieved great success in the field of computer vision, deep neural networks have been introduced into hashing methods, allowing the representation learning part and the binary code learning part to work simultaneously. In deep hashing, the deep features extracted by the network are used to generate binary codes. At the same time, binary code learning provides feedback to guide the representation learning of the deep network.

[0004] However, current image retrieval methods are often based on a large number of training samples, while in reality, the number of available training samples is limited in many scenarios, or the available training resources are insufficient. In addition, the output of deep hashing is limited by discrete constraints. Compared with tasks such as classification tasks where the output is a probability distribution for each category, the binary output retains less semantic information from the original data. Therefore, when faced with scenarios with fewer available training samples, the lack of semantic information makes it difficult for the binary code learning part to guide the deep network to perform representation learning, which further affects the effect of binary code learning. Therefore, a method is needed to overcome the defect of information loss in the deep hashing method and obtain more useful semantic information from limited samples to guide deep hashing for representation learning. Summary of the invention

[0005] The content of this application is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this application is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0006] In view of the problems and shortcomings in the prior art, the present invention aims to provide a deep image retrieval method for small sample scenarios. When the available learning samples are relatively sparse, intermediate continuous features are introduced to mine more information from a small number of samples, guide the deep network to perform representation learning, and then use the trained hash network to calculate the hash code and retrieve it from the database. The efficiency and accuracy of image retrieval are improved, and it is more robust when facing small sample scenarios, so as to solve the problems raised in the above background technology.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] The present invention discloses a deep image retrieval method for small sample scenes, comprising the following steps:

[0009] Step 1: obtain an image dataset H as a database, sample training samples from the database, and calculate the similarity matrix between training samples and between training samples and the database according to the labels of the training sample data;

[0010] Step 2: Generate sample copies by randomly enhancing the training samples, and map them to continuous space through a deep hash network to obtain intermediate features and calculate the contrast loss function L c ;

[0011] Step 3, mapping the learned intermediate representation into a binary code, and calculating a binary loss function in combination with the similarity matrix between the training samples and between the training samples and the database;

[0012] Step 4: Based on the contrast loss function L c Construct the total loss function L with the binary loss function;

[0013] Step 5, updating the deep hash network based on the total loss function L, and updating the hash code of the database sample based on the similarity matrix between the training samples and between the training samples and the database and the total loss function L until the network converges;

[0014] Step 6: For external image samples, use the deep hashing network trained in step 5 above to retrieve the image.

[0015] Furthermore, the specific steps of step 2 are:

[0016] Step 2.1, initializing the deep hash network, wherein the deep hash network consists of a convolutional neural network and a two-layer fully connected network;

[0017] Step 2.2: Perform two random image enhancements on the sampled training samples to obtain two sets of positive and negative sample copies to form the final training data set H. *;

[0018] Step 2.3, use the convolutional neural network to extract features and map them to the l-dimensional space through one layer of the fully connected network to obtain the intermediate feature r i , where the intermediate feature r i Represented as a continuous vector;

[0019] Step 2.4, based on the intermediate feature r i Calculate the contrast loss function L c .

[0020] Furthermore, the specific steps of step 3 are:

[0021] Step 3.1: Use the fully connected network to transform the intermediate feature r i Mapped to vector z i , and then vector z i Converted to binary code and represented as u i =sign(z i ), use tanh(·) to approximate sign(·) to ensure differentiability;

[0022] Step 3.2, based on binary code u i The binary loss function is calculated by the similarity matrix between the training samples and the training samples and the database, where the binary loss function includes the loss function L within the training sample b and the loss function L between the training sample and the database q .

[0023] Furthermore, in step 4, based on the contrast loss function L c The total loss function L is constructed with the binary loss function, and the total loss function L is constructed as L = L c +λ 1 L b +λ 2 L q , where λ 1 ,λ 2 is a hyperparameter.

[0024] Furthermore, the step 5 uses an asymmetric method to directly update the hash code in the database during the training process, and the specific steps are as follows:

[0025] Step 5.1, fix the parameters of the deep hashing network and initialize the binary code V of the database sample;

[0026] Step 5.2: The outputs of the two sets of sample copies generated by the training sample are represented as U 1 and U 2 ,make and will Expand to matrix Step 5.3, simplify the objective function and rewrite it as

[0027]

[0028] in, const means a constant term;

[0029] Step 5.4, using the DCC algorithm to optimize the binary code V of the database sample bit by bit, expressed as:

[0030]

[0031] in, for The kth column of for The remaining matrix after removing the kth column, is the matrix remaining after removing the kth column from V, Q *k is the kth column of Q.

[0032] Furthermore, the step of retrieving the external image sample in step 6 is as follows:

[0033] Step 6.1, obtaining the image sample to be retrieved;

[0034] Step 6.2, load the deep hashing network trained in step 5;

[0035] Step 6.3, calculating the Hamming distance between the binary code of the image sample to be retrieved and the hash code in the database, and sorting them in ascending order;

[0036] Step 6.4, return the image samples corresponding to the top k hash codes.

[0037] Furthermore, the contrast loss function L c It is expressed as:

[0038]

[0039] in, is the number of training samples in this batch whose label is equal to The number of samples, τ is the temperature coefficient, S ij is the similarity matrix between training samples.

[0040] Furthermore, the loss function L in the training sample b Defined as:

[0041]

[0042] Wherein, c is the length of the initialization setting binary code V.

[0043] Furthermore, the loss function L between the training sample and the database is q Defined as:

[0044]

[0045] in, is the similarity matrix between the training sample and the database, Ω i For training samples The corresponding index in the database, c, γ are both hyper parameters.

[0046] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention provides a deep image retrieval method for small sample scenarios, and the specific steps are as follows: first, a small number of training samples are sampled from the database, and the similarity matrix between the training samples and between the training samples and the database is calculated according to the labels; then, positive and negative samples are generated by random image enhancement and mapped to the continuous space to obtain intermediate features and calculate the contrast loss function; the learned intermediate features are mapped to binary codes, and the binary loss function is calculated in combination with the similarity matrix; the deep network model is updated together with the binary loss function and the contrast loss function of the previous step. At the same time, based on the similarity matrix and loss function between the training samples and between the training samples and the database, the hash code of the database sample is updated. Finally, for the foreign image sample, the hash code is calculated using the trained deep hash network, and the hash code in the database is used to retrieve the image. Compared with the existing deep hashing technology, the present invention overcomes the problem that the information loss in binary coding affects the model's representation learning, so that the model can achieve better results even when there are fewer training samples, especially suitable for situations where the training samples are sparse and the training resources are limited. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings constituting a part of this application are used to provide a further understanding of this application, so that other features, purposes and advantages of this application become more obvious. The illustrative embodiment drawings and their descriptions of this application are used to explain this application and do not constitute an improper limitation on this application.

[0048] In the attached picture:

[0049] Figure 1 : It is a schematic diagram of the main steps of the present invention;

[0050] Figure 2 : A schematic diagram of the steps of deep hash network training of the present invention;

[0051] Figure 3 : is a schematic flow chart of the retrieval steps for external image samples to be retrieved in the present invention. DETAILED DESCRIPTION

[0052] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0053] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0054] The present invention discloses a deep image retrieval method for small sample scenes, which will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0055] Reference Figures 1 to 3 As shown, it mainly includes the following steps:

[0056] Step 1: obtain an image dataset H as a database, sample training samples from the database, and calculate the similarity matrix between training samples and between training samples and the database according to the labels of the training sample data;

[0057] Step 2: Generate sample copies by randomly enhancing the training samples, map them to continuous space through the deep hash network to obtain intermediate features and calculate the contrast loss function L c ;

[0058] Step 3: Map the learned intermediate representation into binary code, and calculate the binary loss function by combining the similarity matrix between training samples and between training samples and database;

[0059] Step 4: According to the contrast loss function L c Construct the total loss function L with the binary loss function;

[0060] Step 5: Update the deep hash network based on the total loss function L, and update the hash code of the database sample based on the similarity matrix between training samples and between training samples and database and the total loss function L until the network converges;

[0061] Step 6: For external image samples, use the deep hashing network trained in step 5 above to retrieve the image.

[0062] Specifically, first obtain the image data set H = {(x i ,y i ),i=1,2,…,m} as the database, where xi represents the image vector, y i Indicates the category label to which it belongs, which is a one-hot vector. And initialize the hash code library corresponding to the database sample, including the binary code V={v i ,i=1,2,…,m}, set the length c of the binary code.

[0063] In step 2, the training samples are randomly enhanced to generate sample copies, which are mapped to the continuous space through the deep hash network to obtain the intermediate features and calculate the contrast loss function L c , the specific steps are:

[0064] Step 2.1, initialize the deep hash network, which consists of a convolutional neural network and a two-layer fully connected network;

[0065] Step 2.2: Perform two random image enhancements on the sampled training samples to obtain two sets of positive and negative sample copies to form the final training data set H. * ;

[0066] Step 2.3, use convolutional neural network to extract features and map them to l-dimensional space through one layer of fully connected network to obtain the intermediate feature r i , where the intermediate feature r i Represented as a continuous vector;

[0067] Step 2.4, based on the intermediate feature r i Calculate the contrast loss function L c

[0068] Specifically, N training samples are randomly sampled from the database to represent the available training data and used to form a training sample set. Here, N is usually less than or equal to 200. Then, the sampled training samples x i , and perform two random image enhancements, using methods including cropping, scaling, recoloring, and Gaussian perturbation. Thus, two sets of sample copies are obtained. The corresponding tags meet Construct the final training data set Samples with the same label are positive samples, and samples with different labels are negative samples. We use the improved ResNet50 architecture as the deep hashing model, use the initialized convolutional neural network to extract features, and map them to the l-dimensional space through one of the fully connected layers to obtain the intermediate feature r i , the intermediate feature r i Represented as a continuous vector, where l is set to 128 and can be adjusted according to the data scale. i Calculate the contrast loss function L c , contrast loss function Lc It is expressed as:

[0069]

[0070] in, is the number of training samples in this batch whose label is equal to The number of samples, τ is the temperature coefficient, is the similarity matrix between training samples, when S ij =1 means that the samples belong to the same category.

[0071] In step 3, the learned intermediate representation is mapped into binary code, and the binary loss function is calculated by combining the similarity matrix between training samples and between training samples and the database. The specific steps are as follows:

[0072] Step 3.1: Use a fully connected network to transform the intermediate feature r i Mapped to vector z i , and then vector z i Converted to binary code and represented as u i =sign(z i ), use tanh(·) to approximate sign(·) to ensure differentiability;

[0073] Step 3.2, based on binary code u i The binary loss function is calculated based on the similarity matrix between the training samples and the training samples and the database, where the binary loss function includes the loss function L within the training sample b and the loss function L between the training sample and the database q .

[0074] Specifically, a fully connected network is used to transform the intermediate feature r i Mapped to vector z i , and then vector z i Converted to binary code and represented as u i =sign(z i ). Usually in actual training, we use tanh(·) to approximate sign(·) to ensure that it is differentiable. Based on the binary code u i The binary loss function is calculated by the similarity matrix between the training samples and the training samples and the database. The binary loss function includes the loss function L within the training sample b and the loss function L between the training sample and the database q The loss function L in the training sample b It is expressed as:

[0075]

[0076] Among them, S ijis the similarity matrix between training samples, c is the length of the initialization binary code V, u i is the binary code of the intermediate representation mapping. The loss function L between the training sample and the database q It is expressed as:

[0077]

[0078] in, is the similarity matrix between the training sample and the database, Ω i For training samples The corresponding index in the database, c, γ are both hyper parameters.

[0079] Based on the contrast loss function L c , loss function L within the training sample b and the loss function L between the training sample and the database q Construct the total loss function L, so the total loss function L is constructed as L = L c +λ 1 L b +λ 2 L q , where λ 1 ,λ 2 is a hyperparameter.

[0080] In step 5, the deep hash network is updated based on the total loss function L, and the hash code of the database sample is updated based on the similarity matrix between training samples and between training samples and database and the total loss function L until the network converges. The specific steps include:

[0081] Step 5.1, fix the parameters of the deep hashing network and initialize the binary code V of the database sample;

[0082] Step 5.2: The outputs of the two sets of sample copies generated by the training sample are represented as U 1 and U 2 ,make and will Expand to matrix

[0083] Step 5.3, simplify the objective function and rewrite it as

[0084]

[0085] in, const means a constant term;

[0086] Step 5.4, using the DCC algorithm to optimize the binary code V of the database sample bit by bit, expressed as:

[0087]

[0088] in, for The kth column of for The remaining matrix after removing the kth column, is the matrix remaining after removing the kth column from V, Q *k is the kth column of Q.

[0089] Specifically, The vectors in satisfy For i = 1, 2, ..., n, the remaining vectors are set to 0. The deep hash network is updated using the total loss function L. The hash codes of the database samples are trained and updated based on the similarity matrix between training samples and between training samples and the database and the total loss function L. If the network training update converges, the model and the hash code library in the database are output. If the network training update does not converge, repeat the above steps 1 to 5.

[0090] In step 6, when searching for external image samples, the similarity can be directly calculated based on the hash code library obtained in the training phase, without the need to perform hash operations on the database samples. The specific steps for searching for external image samples are as follows:

[0091] Step 6.1, obtaining the image sample to be retrieved;

[0092] Step 6.2, load the deep hash network model trained in step 5;

[0093] Step 6.3, calculate the Hamming distance between the binary code of the image sample to be retrieved and the hash code in the database, and sort them in ascending order;

[0094] Step 6.4, return the image samples corresponding to the top k hash codes.

[0095] Specifically, first, obtain the image sample to be retrieved; then, load the deep hash network trained in step 5 above, and use it to calculate the binary code of the image sample to be retrieved. Secondly, calculate the Hamming distance of the binary codes in the hash code database (hash code library) of the image sample to be retrieved, and sort the Hamming distance in ascending order. Finally, return the first k binary codes and return their corresponding image samples. Among them, k is set by the system or as needed.

[0096] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A deep image retrieval method for small sample scenes, characterized in that: The steps include: Step 1: obtain an image dataset H as a database, sample training samples from the database, and calculate the similarity matrix between training samples and between training samples and the database according to the labels of the training sample data; Step 2: Generate sample copies by random image enhancement of the training samples, map them to continuous space through a deep hash network to obtain intermediate features and calculate the contrast loss function L c ; Step 3, mapping the learned intermediate features into binary codes, and calculating a binary loss function in combination with the similarity matrix between the training samples and between the training samples and the database; Step 4: According to the contrast loss function L c Construct the total loss function L with the binary loss function; Step 5, updating the deep hash network based on the total loss function L, and updating the hash code of the database sample based on the similarity matrix between the training samples and between the training samples and the database and the total loss function L until the network converges; Step 6: For the external image samples, use the deep hashing network trained in step 5 to retrieve the image; The specific steps of step 2 are: Step 2.1, initializing the deep hash network, which consists of a convolutional neural network and a two-layer fully connected network; Step 2.2: Perform two random image enhancements on the sampled training samples to obtain two sets of positive and negative sample copies, forming the final training data set H * ; Step 2.3, use the convolutional neural network to extract features and map them to the l-dimensional space through one layer of the fully connected network to obtain the intermediate feature r i ; Step 2.4, based on the intermediate feature r i Calculate the contrast loss function L c ; The specific steps of step 3 are: Step 3.1: Use the fully connected network to transform the intermediate feature r i Mapped to vector z i , and then vector z i Convert to binary code u i ; Step 3.2, based on binary code u i The binary loss function is calculated by the similarity matrix between the training samples and the training samples and the database. The binary loss function includes the loss function L within the training sample b and the loss function L between the training sample and the database q ; The step 5 uses an asymmetric method to directly update the hash code in the database during the training process, and the specific steps are: Step 5.1, fix the parameters of the deep hashing network and initialize the binary code V of the database sample; Step 5.2, the two sets of sample copies generated by the training sample are represented as U1 and U2 respectively, and let and will Expand to matrix Step 5.3, simplify the objective function and rewrite it as in, const means a constant term; Step 5.4, using the DCC algorithm to optimize the binary code V of the database sample bit by bit, expressed as in, for The kth column of for The remaining matrix after removing the kth column, is the matrix remaining after removing the kth column from V, Q *k is the kth column of Q.

2. The method for deep image retrieval for small sample scenes according to claim 1, characterized in that: In step 4, based on the contrast loss function L c The total loss function L is constructed with the binary loss function, and the total loss function L is constructed as L = L c +λ1L b +λ2L q , where λ1,λ2 are hyper parameters.

3. The method for deep image retrieval for small sample scenes according to claim 2, characterized in that: The steps for retrieving the external image samples in step 6 are: Step 6.1, obtaining the image sample to be retrieved; Step 6.2, load the deep hash network model trained in step 5; Step 6.3, calculating the Hamming distance between the binary code of the image sample to be retrieved and the hash code in the database, and sorting them in ascending order; Step 6.4, return the image samples corresponding to the top k hash codes.

4. The method for deep image retrieval for small sample scenes according to claim 2, characterized in that: The contrast loss function L c It is expressed as: in, is the number of training samples in this batch whose label is equal to The number of samples, τ is the temperature coefficient, S ij is the similarity matrix between training samples.

5. The method for deep image retrieval for small sample scenes according to claim 4, characterized in that: The loss function L in the training sample b Defined as: Wherein, c is the length of the initialization setting binary code V.

6. The method for deep image retrieval for small sample scenes according to claim 5, characterized in that: The loss function L between the training sample and the database q Defined as: in, is the similarity matrix between the training sample and the database, Ω i For training samples The corresponding index in the database, c, γ are both hyper parameters.

Images