Model training method for images and unsupervised image hash retrieval method

Abstract

The application provides a model training method and an unsupervised image hash retrieval method for images, comprising: obtaining unlabeled training image data; performing twice data augmentation on the unlabeled training image data to obtain first augmented data and second augmented data; performing forward propagation on the first augmented data and the second augmented data respectively by using a pre-constructed hash feature extraction model to obtain first prediction values and second prediction values; calculating a loss value by using a pre-constructed loss function based on the first prediction values and the second prediction values, wherein the loss function is constructed based on a spectral contrast learning algorithm; and updating network parameters in the pre-constructed hash feature extraction model according to the loss value until the loss value is less than a loss threshold to obtain a trained hash feature extraction model. The application can realize unsupervised knowledge transfer, has better generalization ability, focuses on the description of important semantic information, and improves retrieval efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a model training method for images and an unsupervised image hash retrieval method. Background Technology

[0002] Similarity search, also known as near nearest neighbor search, can be content-based image and document search, multimedia retrieval, or plagiarism detection. In the era of big data, similarity search plays a crucial role in modern information retrieval systems.

[0003] In existing technologies, directly searching within the original real-valued feature space incurs extremely high storage and computation costs due to the massive amount of data, thus reducing retrieval efficiency. Hash techniques, however, address this problem by representing data as compact binary code that preserves semantic similarity and calculating the Hamming distance between codes. This significantly reduces memory usage and improves search efficiency, making it a key research focus in recent years.

[0004] Existing hashing methods can be broadly categorized into supervised learning-based and unsupervised learning-based approaches. On the one hand, supervised learning-based hashing relies on data annotation during training, often using metric learning optimization objectives to ensure that semantically similar data have similar binary representation codes, while semantically different data are kept far apart in the Hamming space. Supervised learning-based hashing typically performs well, but its generalization ability is limited. For example, hashing systems trained on face datasets using supervised information often struggle to generalize to fine-grained retrieval tasks on plant datasets.

[0005] On the other hand, many existing competitive unsupervised hashing methods are trained based on the goal of reconstructing data. For example, a binary code is first found by solving a discrete optimization problem of the error between the reconstructed image and the original image. Then, to reduce computational complexity, a variational autoencoder is trained to directly output the hash code. This variational autoencoder is forced to reconstruct the original image from the binary code during training. Similar ideas have also been explored based on generative adversarial networks. It can be seen that the aforementioned unsupervised methods are essentially aiming at data reconstruction, thus requiring the binary code to retain as much information as possible from the original data. However, in hashing tasks, the goal is not to retain all the information from the original data, but rather to retain the semantically relevant information of interest. For example, the image background is useless for similarity searches on face data, but if data reconstruction is the goal, the binary code may contain a lot of background-related information, resulting in a loss of characterization of important semantic information.

[0006] In summary, existing hashing methods based on supervised learning have limited generalization capabilities, while hashing methods based on unsupervised learning produce binary code containing a large amount of invalid and redundant information. Therefore, there is an urgent need for a hashing method that has good generalization performance and can also focus on characterizing important semantic information. Summary of the Invention

[0007] This invention provides a model training method for images and an unsupervised image hash retrieval method to solve the above-mentioned problems.

[0008] This invention provides a model training method for images, comprising:

[0009] Obtain unlabeled training image data;

[0010] The unlabeled training image data is augmented twice to obtain first augmented data and second augmented data;

[0011] The first augmented data and the second augmented data are forward-propagated using a pre-built hash feature extraction model to obtain the first predicted value and the second predicted value.

[0012] Based on the first predicted value and the second predicted value, a loss value is calculated using a pre-constructed loss function, wherein the loss function is constructed based on a spectral contrast learning algorithm;

[0013] The network parameters in the pre-built hash feature extraction model are updated based on the loss value until the loss value is less than the loss threshold, thereby obtaining a trained hash feature extraction model.

[0014] According to the present invention, a model training method for images is provided, wherein the loss function is constructed based on a spectral contrastive learning algorithm, comprising:

[0015] Construct a graph between the first augmented data and the second augmented data, wherein the edges in the graph represent the joint probability distribution of the first augmented data and the second augmented data;

[0016] The loss function is defined using the features output by the pre-built hash feature extraction model. The defined loss function is then used to perform spectral decomposition on the adjacency matrix corresponding to the graph, thereby obtaining the loss function constructed based on the spectral contrastive learning algorithm.

[0017] According to a model training method for images provided by the present invention, the step of constructing a graph between the first augmented data and the second augmented data includes:

[0018] Construct graph A between the first augmented data and the second augmented data;

[0019] Accordingly, the step of using the pre-built hash feature extraction model to extract features and defining a loss function includes:

[0020] Obtain the normalized adjacency matrix corresponding to the graph A.

[0021] The normalized adjacency matrix is ​​based on the matrix factorization criterion. Perform spectral decomposition to obtain the defined loss function l:

[0022]

[0023] Where F∈R N×k Represents the k-dimensional feature matrix of the entire augmented data;

[0024] The feature matrix F is output using a pre-built hash feature extraction model f(x), and the normalized adjacency matrix is ​​then used. Converting to a probability distribution form yields the loss function l′ constructed based on the spectral contrastive learning algorithm:

[0025]

[0026] Where x and x + It is the distribution corresponding to the augmentation of random data. Two independent samples in; x - Unlike unlabeled data Augmented data generated from the data.

[0027] According to a model training method for images provided by the present invention, the step of calculating a loss value based on a first predicted value and a second predicted value using a pre-constructed loss function includes:

[0028] Based on the first predicted value z i =f(x) i ) and the second predicted value z′ i =f(x′) i And based on the loss function l′, a random estimate of the loss function l′ is calculated, and this random estimate is used as the loss value, which is: Where B is the batch size.

[0029] According to the present invention, a model training method for images is provided, wherein the pre-built hash feature extraction model includes a feature extraction backbone network, an activation layer, and a quantization layer;

[0030] Accordingly, the step of using a pre-built hash feature extraction model to perform forward propagation on the first augmented data and the second augmented data respectively to obtain the first predicted value and the second predicted value includes:

[0031] The feature extraction backbone network is used to extract features from the first augmented data and the second augmented data respectively to obtain the first output value and the second output value.

[0032] The first output value and the second output value are converted by the activation layer and the quantization layer respectively to obtain the first predicted value and the second predicted value in binary code form.

[0033] According to a model training method for images provided by the present invention, updating the network parameters in the pre-built hash feature extraction model based on the loss value includes:

[0034] Based on the loss value, and using the pass-through estimator for backpropagation of gradients, the network parameters in the pre-built hash feature extraction model are updated.

[0035] This invention also provides an unsupervised image hash retrieval method, comprising:

[0036] Obtain the image to be retrieved;

[0037] The image to be retrieved is input into a pre-trained hash feature extraction model to obtain the binary code of the image to be retrieved;

[0038] The database images are input into a pre-trained hash feature extraction model to obtain the binary code of the database images. The binary codes of all database images constitute a binary code library.

[0039] Calculate the Hamming distance between the binary code of the image to be retrieved and each binary code in the binary code library, and take the database image with the smallest Hamming distance as the retrieval result of the image to be retrieved;

[0040] The pre-trained hash feature extraction model is trained using any of the above-described image model training methods.

[0041] The present invention also provides a model training apparatus for images, comprising:

[0042] The training image acquisition module is used to acquire unlabeled training image data;

[0043] The data augmentation module performs two data augmentations on the unlabeled training image data to obtain first augmented data and second augmented data;

[0044] The forward propagation module is used to perform forward propagation on the first augmented data and the second augmented data respectively using a pre-built hash feature extraction model to obtain the first predicted value and the second predicted value.

[0045] The loss value calculation module is used to calculate the loss value based on the first predicted value and the second predicted value using a pre-constructed loss function, wherein the loss function is constructed based on the spectral contrast learning algorithm;

[0046] The model update module is used to update the network parameters in the pre-built hash feature extraction model according to the loss value, until the loss value is less than the loss threshold and the update of the network parameters is stopped, so as to obtain the trained hash feature extraction model.

[0047] The present invention also provides an unsupervised image hash retrieval device, comprising:

[0048] The image acquisition module is used to acquire the image to be retrieved.

[0049] The binary code acquisition module is used to input the image to be retrieved into a pre-trained hash feature extraction model to obtain the binary code of the image to be retrieved.

[0050] The binary code library acquisition module is used to input the database images into a pre-trained hash feature extraction model to obtain the binary code of the database images. The binary codes of all database images constitute the binary code library.

[0051] The retrieval result determination module is used to calculate the Hamming distance between the binary code of the image to be retrieved and each binary code in the binary code library, and to take the database image with the smallest Hamming distance as the retrieval result of the image to be retrieved.

[0052] The pre-trained hash feature extraction model is trained using the aforementioned image model training device.

[0053] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements any of the above-described model training methods or unsupervised image hash retrieval methods for images.

[0054] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the above-described image model training methods or unsupervised image hash retrieval methods.

[0055] This invention provides a model training method and an unsupervised image hash retrieval method for images. The image training method involves performing two data augmentations on the unlabeled training image data to obtain first augmented data and second augmented data. Based on these first and second augmented data, corresponding first and second predicted values ​​are obtained. A loss value is calculated using a loss function constructed based on a spectral contrastive learning algorithm, and this loss value is used to train a hash feature extraction model. This invention, through spectral contrastive learning, learns a binary representation that is far superior to hash methods based on data reconstruction. When the dataset for unsupervised training is large enough, the hash feature extraction model f trained by this invention can be flexibly and efficiently applied to various downstream tasks, achieving unsupervised knowledge transfer, exhibiting better generalization ability, emphasizing the characterization of important semantic information, improving retrieval efficiency, and enhancing the effectiveness of image retrieval, face recognition, and other systems. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0057] Figure 1 This is a schematic flowchart of a model training method for images provided in an embodiment of the present invention;

[0058] Figure 2 This is a flowchart illustrating the unsupervised image hash retrieval method provided in an embodiment of the present invention;

[0059] Figure 3 This is one of the schematic diagrams of dataset verification results provided in the embodiments of the present invention;

[0060] Figure 4 This is the second schematic diagram of the dataset verification results provided in the embodiments of the present invention;

[0061] Figure 5 A schematic diagram of the structure of a model training device for images provided in an embodiment of the present invention;

[0062] Figure 6 This is a schematic diagram of the unsupervised image hash retrieval device provided in an embodiment of the present invention;

[0063] Figure 7 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0065] Before introducing the model training method for images and the unsupervised image hash retrieval method of the present invention, the main concepts of the present invention will be explained first.

[0066] Contrastive learning is a self-supervised learning method used to learn general features of a dataset by having the model learn which data points are similar or different, even without labels. By distinguishing between instances through contrastive learning, the learned feature representations have good discriminative power, thereby deepening the characterization of important semantic information.

[0067] This invention enables a generalized and reliable unsupervised image hashing method using contrastive learning. From another perspective, this invention can also be seen as an effective extension of the classic spectral hashing method in the field of deep learning. Hash representation aims to find binary data representations such that the Hamming distance between them reflects the semantic similarity between the original data. The spectral hashing method represents hash learning as a spectral decomposition problem of a kernel function defined on a graph adjacency matrix. Therefore, it uses the eigenfunctions of a specific kernel function to capture semantic similarity, generate continuous representations, and then applies quantization to obtain binary codewords. However, spectral hashing relies on typical kernel functions because this allows the use of their corresponding analytical eigenfunctions, thus limiting its processing to simple Euclidean data and presenting problems when dealing with complex data such as images.

[0068] Furthermore, existing self-supervised learning methods define different learning tasks to achieve representation learning. For example, they model the similarity between image pairs through contrastive or non-contrastive learning objectives. However, due to the existence of trivial solutions, existing self-supervised learning methods often introduce specific techniques, such as large batches, asymmetric mechanisms, and momentum encoders, to prevent representation collapse. In contrast, Spectral Contrastive Learning (SCL) eliminates the dependence on these techniques by establishing a connection between contrastive learning and spectral decomposition, thus contributing to more reliable and interpretable unsupervised learning. SCL is also a method for learning eigenfunctions defined on a normalized adjacency matrix kernel function over high-dimensional complex data; therefore, SCL can be used to effectively extend spectral hashing in the deep learning era.

[0069] The model training method for images and the unsupervised image hash retrieval method in this application will be described in detail below with reference to the accompanying drawings.

[0070] Figure 1 This is a schematic flowchart of a model training method for images provided in an embodiment of the present invention; as shown below. Figure 1 As shown, the model training method for images includes:

[0071] S101, Obtain unlabeled training image data.

[0072] In this step, we first obtain a clean dataset. (N is the total number of data points), this clean dataset consists of unlabeled images, from which a batch of data is randomly selected. (B is the batch size) is used as unlabeled training image data.

[0073] The clean dataset can be determined based on the application of image hash retrieval. For example, if image hash retrieval is applied to face retrieval, the clean dataset consists of unlabeled face images; if image hash retrieval is applied to product retrieval, the clean dataset consists of unlabeled products of various types. This invention does not limit this.

[0074] S102, perform two data augmentations on the unlabeled training image data to obtain first augmented data and second augmented data. That is, augment the unlabeled training image data... Each unlabeled training image in the dataset undergoes two data augmentations to obtain the first augmented data x. i And the second augmented data x′ i x i With x′ i Satisfy x i ,

[0075] S103, using the pre-built hash feature extraction model f to process the first augmented data x respectively. i And the second augmented data x′ i Perform forward propagation to obtain the first predicted value z. i =f(x) i ) and the second predicted value z′ i =f(x′) i ).

[0076] The pre-built hash feature extraction model f is a neural network model with binary code output, which can be any conventional neural network, such as AlexNet, GooleNet, VGG, ResNet, DenseNet, etc.

[0077] S104, based on the first predicted value z i and the second predicted value z′ i The loss value L is calculated using a pre-constructed loss function.

[0078] The loss function l′ is constructed based on the spectral contrast learning algorithm. This loss function l′ enables the hash feature extraction model f to learn the first augmented data x. i With the second augmented data x′ i The similarities or differences between instances are used to distinguish between them; the specific construction process is described below.

[0079] S105, update the network parameters in the pre-built hash feature extraction model f according to the loss value L, and stop updating the network parameters until the loss value L is less than the loss threshold, so as to obtain the trained hash feature extraction model.

[0080] In this step, if the loss value L does not meet the loss threshold requirement, the dataset is restarted from a clean dataset. A batch of data was randomly selected as unlabeled training image data. Then, data augmentation is performed, and S101-S105 are executed repeatedly to update the network parameters of the hash feature extraction model f.

[0081] The image model training method provided in this invention involves performing two data augmentations on the unlabeled training image data to obtain first augmented data and second augmented data. Based on these two augmented data, corresponding first and second predicted values ​​are obtained. A loss value is calculated using a loss function constructed based on a spectral contrastive learning algorithm, and this loss value is used to train a hash feature extraction model. This invention, through spectral contrastive learning, learns a binary representation that is far superior to hash methods based on data reconstruction. When the unsupervised training dataset is large enough, the hash feature extraction model f trained by this invention can be flexibly and efficiently applied to various downstream tasks, achieving unsupervised knowledge transfer, exhibiting better generalization ability, emphasizing the characterization of important semantic information, improving retrieval efficiency, and enhancing the effectiveness of image retrieval, face recognition, and other systems.

[0082] In some embodiments of the present invention, the loss function is constructed based on a spectral contrastive learning algorithm, including:

[0083] Construct a graph between the first augmented data and the second augmented data, wherein the edges in the graph represent the joint probability distribution of the first augmented data and the second augmented data.

[0084] The loss function is defined using the features output by the pre-built hash feature extraction model. The defined loss function is then used to perform spectral decomposition on the adjacency matrix corresponding to the graph, thereby obtaining the loss function constructed based on the spectral contrastive learning algorithm.

[0085] Specifically, obtain the first augmented data x i With the second augmented data x′ i Graph A is between these two graphs. Here, the Spectral Contrast Learning (SCL) algorithm defines a graph A based on random data augmentation, which first acquires unlabeled images. (i.e., unlabeled training image data) Distribution of a frame of an image For unlabeled images For data augmentation, let the distribution corresponding to random data augmentation be denoted as . The distribution of the augmented data is then as follows: In unlabeled images After performing two data augmentations to obtain the first augmented image x and the second augmented image x′, the joint probability distribution between the first augmented image x and the second augmented image x′ is as follows: The joint probability p(x,x′) describes the probability of generating x and x′ from the same clean data through random augmentation, which can be viewed here as a measure of the semantic similarity between x and x′. SCL is therefore defined to obtain a graph A on these augmented data, with weights A on graph A. x,x′ = p(x,x′), where there are infinitely many nodes in graph A, and nodes with similar semantics are connected to each other.

[0086] After obtaining graph A, SCL then attempts to learn the spectral information of the graph. This is achieved by performing spectral decomposition on the adjacency matrix corresponding to the graph, specifically including:

[0087] Obtain the normalized adjacency matrix corresponding to the graph A. That is, the normalized adjacency matrix transformation of Graph A is performed using the following formula:

[0088]

[0089] Where D is a diagonal matrix, and D x,x = p(x).

[0090] The normalized adjacency matrix is ​​based on the matrix factorization criterion. Perform spectral decomposition to obtain the defined loss function l:

[0091]

[0092] Where F∈R ×kLet represent the k-dimensional feature matrix of all augmented data (including the first augmented data and the second augmented data).

[0093] The defined loss function is used to perform spectral decomposition on the adjacency matrix corresponding to the graph, thereby obtaining the loss function constructed based on the spectral contrastive learning algorithm, including:

[0094] The k-dimensional feature matrix F output by the pre-constructed hash feature extraction model f(x) is used, and the normalized adjacency matrix is... Converting to a probability distribution form yields the loss function l′ constructed based on the spectral contrastive learning algorithm:

[0095]

[0096] Where x and x + It is the distribution corresponding to the augmentation of random data. Two independent samples in; x - Unlike unlabeled data Augmented data generated from the data.

[0097] The image model training method provided in this invention effectively transforms spectral decomposition on augmented images into a concise contrastive learning objective for neural networks using the SCL algorithm. While maintaining theoretical assurance, it also enables hash feature extraction models to benefit from the effectiveness of contrastive learning.

[0098] In some embodiments of the present invention, the step of calculating the loss value based on the first predicted value and the second predicted value using a pre-constructed loss function includes:

[0099] Based on the first predicted value z i =f(x) i ) and the second predicted value z i ′=f(x i Based on the loss function l′, a random estimate of the loss function l′ is calculated, and this random estimate is used as the loss value L. Where B is the batch size.

[0100] Since Figure A above is based on a first augmented image x and a second augmented image x′, therefore, in the first augmented data x i And the second augmented data x i Based on this, the loss calculated by the loss function l′ needs to be randomly estimated according to the batch size B, so as to obtain the final loss value L.

[0101] The image model training method provided in this embodiment of the invention calculates the loss based on the loss function l′ described above, performs random estimation on the loss calculation result, and uses the random estimation result as the final loss value to update the hash feature extraction model.

[0102] In some embodiments of the present invention, in addition to optimizing the loss function l′, an additional constraint is introduced, namely: the neural approximation of the true feature function is binary. Specifically, the pre-built hash feature extraction model includes a feature extraction backbone network, an activation layer, and a quantization layer.

[0103] Accordingly, the step of using a pre-built hash feature extraction model to perform forward propagation on the first augmented data and the second augmented data respectively to obtain the first predicted value and the second predicted value includes:

[0104] The feature extraction backbone network extracts features from the first augmented data and the second augmented data respectively, obtaining a first output value and a second output value, which are then considered as sigmoidlogits. It should be noted that the first output value and the second output value are the original outputs.

[0105] The first output value and the second output value are transformed by the activation layer (i.e., the sigmoid layer) and the quantization layer respectively to obtain the first predicted value and the second predicted value in binary code form. The first predicted value and the second predicted value in binary code form are used as the final output of the model.

[0106] The image model training method provided in this embodiment of the invention enables the model to output a first predicted value and a second predicted value in binary form by adding an activation layer and a quantization layer after the feature extraction backbone network.

[0107] In some embodiments of the present invention, updating the network parameters in the pre-built hash feature extraction model based on the loss value includes:

[0108] Based on the loss value, and using the straight-through estimator for backpropagation of gradients, the network parameters in the pre-built hash feature extraction model are updated.

[0109] Figure 2 This is a flowchart illustrating the unsupervised image hash retrieval method provided in an embodiment of the present invention, as shown below. Figure 2 As shown, the unsupervised image hash retrieval method includes:

[0110] S201, Obtain the image to be retrieved y* ;

[0111] S202, the image to be retrieved y * The binary code b of the image to be retrieved is obtained by inputting the pre-trained hash feature extraction model f into the model f. * ;

[0112] S203, transfer the database image The binary codes of the database images are input into a pre-trained hash feature extraction model f to obtain the binary codes of all the database images. The binary codes of all the database images constitute a binary code library.

[0113] S204, Calculate the binary code b of the image to be retrieved. * With binary code library The Hamming distance between each binary code in the database is used to determine the image in the database with the smallest Hamming distance as the retrieval result.

[0114] The pre-trained hash feature extraction model f is trained using the above-described model training method for images.

[0115] In this embodiment, the hash feature extraction model f is a model based on the ResNet-50 network structure and is trained on the ImageNet dataset. Then, experiments on unsupervised image hash retrieval methods are conducted using the COCO dataset.

[0116] The validation results on the COCO dataset are as follows: Figure 3 And as shown in 4, Figure 3 and Figure 4 The first column of images in the table consists of images to be retrieved, while the remaining ten columns are the retrieval results. The five rows of retrieval results correspond to the results retrieved using 4-dimensional, 8-dimensional, 16-dimensional, 32-dimensional, and 64-dimensional hash codes, respectively. Figure 3 The image to be searched is a woman sitting on a beach. The first row of 4D search results shows the element of "people," but the elements such as quantity, gender, and location do not match the image. The second row of 8D search results is similar to the 4D results. In the third and fourth rows, the elements such as location, gender, and number of people are quite similar to the image. The fifth row of search results shows a high degree of overlap with the beach background, and the number of people mostly matches. Similarly, Figure 4 The image to be retrieved is a bus traveling at night. The search results in the first two rows are quite different from the image to be retrieved. In the third and fourth rows, the number of buses increases, and in the last fifth row, many images are highly similar to the image to be retrieved. This shows that the higher the dimension of the hash code, the more accurate the retrieved results.

[0117] In addition, the unsupervised image hash retrieval method of the present invention was verified on the NUS-WIDE dataset and the ImageNet dataset. The specific mAP and precision are shown in Table 1.

[0118] Table 1 Performance Comparison of Unsupervised Image Hash Retrieval Methods

[0119]

[0120] As shown in Table 1, the mAP and precision both continuously improve with the increase of hash code dimension. In terms of performance across various datasets, the unsupervised image hashing retrieval method of this invention performs best on the NUS-WIDE dataset, followed by the COCO dataset, while its performance on the ImageNet dataset is relatively poor.

[0121] In summary, the unsupervised image hash retrieval method of the present invention has good retrieval accuracy.

[0122] The following describes the image model training apparatus and the unsupervised image hash retrieval apparatus provided by the present invention. The image model training apparatus described below and the image model training method described above can be referred to in correspondence with each other. The unsupervised image hash retrieval apparatus and the unsupervised image hash retrieval method can be referred to in correspondence with each other.

[0123] Figure 5 This is a schematic diagram of the structure of a model training device for images provided in an embodiment of the present invention, as shown below. Figure 5 As shown, the model training device for images includes a training image acquisition module 501, a data augmentation module 502, a forward propagation module 503, a loss value calculation module 504, and a model update module 505.

[0124] The training image acquisition module 501 is used to acquire unlabeled training image data.

[0125] In this module, we first obtain a clean dataset. (N is the total number of data points), this clean dataset consists of unlabeled images, from which a batch of data is randomly selected. (B is the batch size) is used as unlabeled training image data.

[0126] The clean dataset can be determined based on the application of image hash retrieval. For example, if image hash retrieval is applied to face retrieval, the clean dataset consists of unlabeled face images; if image hash retrieval is applied to product retrieval, the clean dataset consists of unlabeled products of various types. This invention does not limit this.

[0127] The data augmentation module 502 performs two data augmentations on the unlabeled training image data to obtain first augmented data and second augmented data. That is, it augments the unlabeled training image data... Each unlabeled training image in the dataset undergoes two data augmentations to obtain the first augmented data x. i And the second augmented data x′ i x i With x′ i Satisfy x i ,

[0128] Forward propagation module 503 is used to process the first augmented data x using a pre-built hash feature extraction model f. i And the second augmented data x′ i Perform forward propagation to obtain the first predicted value z. i =f(x) i ) and the second predicted value z′ i =f(x′) i ).

[0129] The pre-built hash feature extraction model f is a neural network model with binary code output, which can be any conventional neural network, such as AlexNet, GooleNet, VGG, ResNet, DenseNet, etc.

[0130] Loss calculation module 504 is used to calculate the loss value based on the first predicted value z. i And the second predicted value z i The loss value L is calculated using a pre-constructed loss function.

[0131] The loss function l′ is constructed based on the spectral contrast learning algorithm. This loss function l′ enables the hash feature extraction model f to learn the first augmented data x. i With the second augmented data x i The similarities or differences between ' are used to distinguish between instances; see the description above for the specific construction process.

[0132] The model update module 505 is used to update the network parameters in the pre-built hash feature extraction model f according to the loss value L, until the loss value L is less than the loss threshold and the update of the network parameters is stopped, so as to obtain the trained hash feature extraction model.

[0133] In this module, if the loss value L does not meet the loss threshold, the dataset is re-processed from a clean dataset. A batch of data was randomly selected as unlabeled training image data. Then, data augmentation is performed, and S101-S105 are executed repeatedly to update the network parameters of the hash feature extraction model f.

[0134] The image model training apparatus provided in this invention performs two data augmentations on the unlabeled training image data to obtain first augmented data and second augmented data. Based on the first and second augmented data, corresponding first and second predicted values ​​are obtained. A loss value is calculated using a loss function constructed based on a spectral contrastive learning algorithm, and this loss value is used to train a hash feature extraction model. This invention, through spectral contrastive learning, learns a binary representation that is far superior to hash methods based on data reconstruction. When the dataset for unsupervised training is large enough, the hash feature extraction model f trained by this invention can be flexibly and efficiently applied to various downstream tasks, achieving unsupervised knowledge transfer, exhibiting better generalization ability, emphasizing the characterization of important semantic information, improving retrieval efficiency, and enhancing the effectiveness of image retrieval, face recognition, and other systems.

[0135] In some embodiments of the present invention, the loss value calculation module 504 includes a loss function construction unit, which includes a data graph construction subunit and a loss function construction subunit.

[0136] A data graph construction subunit is used to construct a graph between the first augmented data and the second augmented data, wherein the edges in the graph represent the joint probability distribution of the first augmented data and the second augmented data.

[0137] The loss function construction subunit is used to define the loss function using the features output by the pre-built hash feature extraction model, and to perform spectral decomposition on the adjacency matrix corresponding to the graph using the defined loss function, thereby obtaining the loss function constructed based on the spectral contrastive learning algorithm.

[0138] Specifically, obtain the first augmented data x i With the second augmented data x′ i Graph A is between these two graphs. Here, the Spectral Contrast Learning (SCL) algorithm defines a graph A based on random data augmentation, which first acquires unlabeled images. (i.e., unlabeled training image data) Distribution of a frame of an image For unlabeled images For data augmentation, let the distribution corresponding to random data augmentation be denoted as . The distribution of the augmented data is then as follows: In unlabeled images After performing two data augmentations to obtain the first augmented image x and the second augmented image x′, the joint probability distribution between the first augmented image x and the second augmented image x′ is as follows: The joint probability p(x,x′) describes the probability of generating x and x′ from the same clean data through random augmentation, which can be viewed here as a measure of the semantic similarity between x and x′. SCL is therefore defined to obtain a graph A on these augmented data, with weights A on graph A. x,x′ = p(x,x′), where there are infinitely many nodes in graph A, and nodes with similar semantics are connected to each other.

[0139] After obtaining graph A, SCL then attempts to learn the spectral information of the graph. This is achieved by performing spectral decomposition on the adjacency matrix corresponding to the graph, specifically including:

[0140] Obtain the normalized adjacency matrix corresponding to the graph A. That is, the normalized adjacency matrix transformation of Graph A is performed using the following formula:

[0141]

[0142] Where D is a diagonal matrix, and D x,x = p(x).

[0143] The normalized adjacency matrix is ​​based on the matrix factorization criterion. Perform spectral decomposition to obtain the defined loss function l:

[0144]

[0145] Where F∈R ×k Let represent the k-dimensional feature matrix of all augmented data (including the first augmented data and the second augmented data).

[0146] The defined loss function is used to perform spectral decomposition on the adjacency matrix corresponding to the graph, thereby obtaining the loss function constructed based on the spectral contrastive learning algorithm, including:

[0147] The k-dimensional feature matrix F output by the pre-constructed hash feature extraction model f(x) is used, and the normalized adjacency matrix is... Converting to a probability distribution form yields the loss function l′ constructed based on the spectral contrastive learning algorithm:

[0148]

[0149] Where x and x + It is the distribution corresponding to the augmentation of random data. Two independent samples in; x - Unlike unlabeled data Augmented data generated from the data.

[0150] The image model training apparatus provided in this invention effectively transforms spectral decomposition on augmented images into a concise contrastive learning objective for neural networks using the SCL algorithm. While maintaining theoretical assurance, it also enables hash feature extraction models to benefit from the effectiveness of contrastive learning.

[0151] In some embodiments of the present invention, the loss value calculation module 504 includes a loss value calculation unit.

[0152] The loss value calculation unit is used to calculate the loss value based on the first predicted value z. i =f(x) i ) and the second predicted value z i ′=f(x i Based on the loss function l′, a random estimate of the loss function l′ is calculated, and this random estimate is used as the loss value L. Where B is the batch size.

[0153] Since Figure A above is based on a first augmented image x and a second augmented image x′, therefore, in the first augmented data x i And the second augmented data x i Based on this, the loss calculated by the loss function l′ needs to be randomly estimated according to the batch size B, so as to obtain the final loss value L.

[0154] The image model training device provided in this embodiment of the invention performs loss calculation based on the loss function l′ described above, performs random estimation calculation on the loss calculation result, and uses the random estimation result as the final loss value, thereby updating the hash feature extraction model.

[0155] In some embodiments of the present invention, in addition to optimizing the loss function l′, an additional constraint is introduced, namely: the neural approximation of the true feature function is binary. Specifically, the pre-built hash feature extraction model includes a feature extraction backbone network, an activation layer, and a quantization layer.

[0156] Accordingly, the step of using a pre-built hash feature extraction model to perform forward propagation on the first augmented data and the second augmented data respectively to obtain the first predicted value and the second predicted value includes:

[0157] The feature extraction backbone network extracts features from the first augmented data and the second augmented data respectively, obtaining a first output value and a second output value, which are then considered as sigmoidlogits. It should be noted that the first output value and the second output value are the original outputs.

[0158] The first output value and the second output value are transformed by the activation layer (i.e., the sigmoid layer) and the quantization layer respectively to obtain the first predicted value and the second predicted value in binary code form. The first predicted value and the second predicted value in binary code form are used as the final output of the model.

[0159] The image model training apparatus provided in this embodiment of the invention enables the model to output a first predicted value and a second predicted value in binary form by adding an activation layer and a quantization layer after the feature extraction backbone network.

[0160] In some embodiments of the present invention, the model update module 505 includes a model update unit.

[0161] The model update unit is used to update the network parameters in the pre-built hash feature extraction model based on the loss value and by backpropagating the gradient using a straight-through estimator.

[0162] Figure 6 This is a schematic diagram of the unsupervised image hash retrieval device provided in an embodiment of the present invention, as shown below. Figure 6 As shown, the unsupervised image hash retrieval device includes a module 601 for acquiring the image to be retrieved, a module 602 for acquiring binary code, a module 603 for acquiring binary code library, and a module 604 for determining retrieval results.

[0163] Image acquisition module 601 is used to acquire the image to be retrieved y. * ;

[0164] Binary code acquisition module 602 is used to obtain the image to be retrieved y * The binary code b of the image to be retrieved is obtained by inputting the pre-trained hash feature extraction model f into the model f. * ;

[0165] Binary code library acquisition module 603 is used to retrieve images from the database. The binary codes of the database images are input into a pre-trained hash feature extraction model f to obtain the binary codes of all the database images. The binary codes of all the database images constitute a binary code library.

[0166] The retrieval result determination module 604 is used to calculate the binary code b of the image to be retrieved. * With binary code library The Hamming distance between each binary code in the database is used to determine the image in the database with the smallest Hamming distance as the retrieval result.

[0167] The pre-trained hash feature extraction model f is trained using the above-described model training method for images.

[0168] The unsupervised image hash retrieval device provided in this embodiment of the invention has a good retrieval accuracy.

[0169] Figure 7 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as... Figure 7 As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, communications interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a model training method for images. This model training method for images includes: acquiring unlabeled training image data; performing two data augmentations on the unlabeled training image data to obtain first augmented data and second augmented data; performing forward propagation on the first augmented data and the second augmented data respectively using a pre-built hash feature extraction model to obtain a first predicted value and a second predicted value; calculating a loss value based on the first predicted value and the second predicted value using a pre-built loss function, wherein the loss function is constructed based on a spectral contrastive learning algorithm; updating the network parameters in the pre-built hash feature extraction model according to the loss value until the loss value is less than a loss threshold, thereby obtaining a trained hash feature extraction model.

[0170] Alternatively, it can be used to perform an unsupervised image hash retrieval method, which includes: acquiring an image to be retrieved; inputting the image to be retrieved into a pre-trained hash feature extraction model to obtain the binary code of the image to be retrieved; inputting database images into a pre-trained hash feature extraction model to obtain the binary code of the database images, wherein the binary codes of all database images constitute a binary code library; calculating the Hamming distance between the binary code of the image to be retrieved and each binary code in the binary code library, and taking the database image with the smallest Hamming distance as the retrieval result of the image to be retrieved; wherein the pre-trained hash feature extraction model is trained using any of the above-described image model training methods.

[0171] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0172] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a model training method for images. This model training method for images includes: acquiring unlabeled training image data; performing two data augmentations on the unlabeled training image data to obtain first augmented data and second augmented data; performing forward propagation on the first augmented data and the second augmented data respectively using a pre-constructed hash feature extraction model to obtain a first predicted value and a second predicted value; calculating a loss value based on the first predicted value and the second predicted value using a pre-constructed loss function, wherein the loss function is constructed based on a spectral contrastive learning algorithm; and updating the network parameters in the pre-constructed hash feature extraction model according to the loss value until the loss value is less than a loss threshold, thereby obtaining a trained hash feature extraction model.

[0173] Alternatively, it can be used to perform an unsupervised image hash retrieval method, which includes: acquiring an image to be retrieved; inputting the image to be retrieved into a pre-trained hash feature extraction model to obtain the binary code of the image to be retrieved; inputting database images into a pre-trained hash feature extraction model to obtain the binary code of the database images, wherein the binary codes of all database images constitute a binary code library; calculating the Hamming distance between the binary code of the image to be retrieved and each binary code in the binary code library, and taking the database image with the smallest Hamming distance as the retrieval result of the image to be retrieved; wherein the pre-trained hash feature extraction model is trained using any of the above-described image model training methods.

[0174] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0175] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A model training method for images, characterized in that, include: Obtain unlabeled training image data; The unlabeled training image data is augmented twice to obtain first augmented data and second augmented data; The first augmented data and the second augmented data are forward-propagated using a pre-built hash feature extraction model to obtain the first predicted value and the second predicted value. Based on the first predicted value and the second predicted value, a loss value is calculated using a pre-constructed loss function, including: based on the first predicted value and the second predicted value, a random estimate of the loss function is calculated according to the loss function, and the random estimate is used as the loss value, wherein the loss function is constructed based on a spectral contrastive learning algorithm; The network parameters in the pre-built hash feature extraction model are updated according to the loss value until the loss value is less than the loss threshold, so as to obtain a trained hash feature extraction model. The pre-built hash feature extraction model includes a feature extraction backbone network, an activation layer, and a quantization layer. Accordingly, the step of using a pre-built hash feature extraction model to perform forward propagation on the first augmented data and the second augmented data respectively to obtain the first predicted value and the second predicted value includes: The feature extraction backbone network is used to extract features from the first augmented data and the second augmented data respectively to obtain the first output value and the second output value. The first output value and the second output value are converted by the activation layer and the quantization layer respectively to obtain the first predicted value and the second predicted value in binary code form.

2. The model training method for images according to claim 1, characterized in that, The loss function is constructed based on the spectral contrastive learning algorithm and includes: Construct a graph between the first augmented data and the second augmented data, wherein the edges in the graph represent the joint probability distribution of the first augmented data and the second augmented data; The loss function is defined using the features output by the pre-built hash feature extraction model. The defined loss function is then used to perform spectral decomposition on the adjacency matrix corresponding to the graph, thereby obtaining the loss function constructed based on the spectral contrastive learning algorithm.

3. The model training method for images according to claim 2, characterized in that, The construction of the graph between the first augmented data and the second augmented data includes: Construct graph A between the first augmented data and the second augmented data; Accordingly, the step of using the pre-built hash feature extraction model to extract features and defining a loss function includes: Obtain the normalized adjacency matrix corresponding to the graph A. ; The normalized adjacency matrix is ​​based on the matrix factorization criterion. Perform spectral decomposition to obtain the defined loss function. : ； Among them, F Represents the k-dimensional feature matrix of the entire augmented data; Utilizing a pre-built hash feature extraction model Output the feature matrix F, and then use the normalized adjacency matrix... Converting to a probability distribution form yields the loss function constructed based on the spectral contrastive learning algorithm. : ； Where x and It is the distribution p(x| corresponding to the augmentation of random data. Two independent samples in ); Unlike unlabeled data Augmented data generated from the data.

4. The model training method for images according to claim 3, characterized in that, The step of calculating the loss value based on the first predicted value and the second predicted value using a pre-constructed loss function includes: Based on the first predicted value and the second predicted value And according to the loss function The loss function is calculated. The random estimate is used as the loss value, which is: ； Where B is the batch size.

5. The model training method for images according to claim 1, characterized in that, The step of updating the network parameters in the pre-built hash feature extraction model based on the loss value includes: Based on the loss value, and using the pass-through estimator for backpropagation of gradients, the network parameters in the pre-built hash feature extraction model are updated.

6. An unsupervised image hashing retrieval method, characterized in that, include: Obtain the image to be retrieved; The image to be retrieved is input into a pre-trained hash feature extraction model to obtain the binary code of the image to be retrieved; The database images are input into a pre-trained hash feature extraction model to obtain the binary code of the database images. The binary codes of all database images constitute a binary code library. Calculate the Hamming distance between the binary code of the image to be retrieved and each binary code in the binary code library, and take the database image with the smallest Hamming distance as the retrieval result of the image to be retrieved; The pre-trained hash feature extraction model is trained using the image model training method described in any one of claims 1-5.

7. A model training device for images, characterized in that, include: The training image acquisition module is used to acquire unlabeled training image data; The data augmentation module performs two data augmentations on the unlabeled training image data to obtain first augmented data and second augmented data; The forward propagation module is used to perform forward propagation on the first augmented data and the second augmented data respectively using a pre-built hash feature extraction model to obtain the first predicted value and the second predicted value. The loss value calculation module is used to calculate a loss value based on the first predicted value and the second predicted value using a pre-constructed loss function, including: calculating a random estimate of the loss function based on the first predicted value and the second predicted value, and using the random estimate as the loss value, wherein the loss function is constructed based on a spectral contrastive learning algorithm; The model update module is used to update the network parameters in the pre-built hash feature extraction model according to the loss value, until the loss value is less than the loss threshold and the update of the network parameters is stopped, so as to obtain the trained hash feature extraction model. The pre-built hash feature extraction model includes a feature extraction backbone network, an activation layer, and a quantization layer. Accordingly, the step of using a pre-built hash feature extraction model to perform forward propagation on the first augmented data and the second augmented data respectively to obtain the first predicted value and the second predicted value includes: The feature extraction backbone network is used to extract features from the first augmented data and the second augmented data respectively to obtain the first output value and the second output value. The first output value and the second output value are converted by the activation layer and the quantization layer respectively to obtain the first predicted value and the second predicted value in binary code form.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the model training method for images as described in any one of claims 1 to 5 or the unsupervised image hash retrieval method as described in claim 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the model training method for images as described in any one of claims 1 to 5 or the unsupervised image hash retrieval method as described in claim 6.

Images