Image recognition method, system and equipment for prototype weighted contrast learning and medium

Abstract

The invention relates to the field of machine learning, in particular to an image recognition method, system and device for prototype weighted contrast learning and a medium, and the method comprises the steps: obtaining sample feature information, and carrying out the clustering of the sample feature information, and obtaining first clustering information; constructing a clustering consistency loss function according to the first clustering information of the two pieces of sample feature information to obtain a clustering result; optimizing the clustering consistency loss function, acquiring negative sample feature information under the condition that the clustering result is judged to be credible, and clustering the negative sample feature information to obtain second clustering information; constructing a weighted comparison learning loss function according to the first clustering information and the second clustering information to obtain a comparison result; and constructing a total loss function according to a clustering result and a comparison result to obtain total loss information for updating model parameters by gradient back propagation, accurately distinguishing similar samples, reducing the use of resources and time during learning, and improving the accuracy of image recognition.

Description

Technical Field

[0001] The present application relates to the field of machine learning, and in particular to an image recognition method, system, device and medium for prototype weighted contrast learning. Background Art

[0002] In the field of image recognition, accurately classifying and identifying images is the core of many applications, such as autonomous driving, security monitoring, and medical image analysis. Traditional image recognition methods rely on large amounts of manually annotated datasets, which is not only costly but also time-consuming. Self-supervised learning has become a powerful tool for solving many real-world problems because it does not require manually annotated labels, especially when the amount of data is large and the annotation cost is high. Self-supervised learning is a tool that can achieve large amounts of data without manually annotating labels. Common methods of self-supervised learning include generative methods, predictive methods, and contrastive learning. Generative methods learn by reconstructing the original data, such as generative adversarial networks. Predictive methods learn by letting the model predict part of the data, such as predicting image transformations. Contrastive learning learns by comparing the differences between different samples. Specifically, contrastive learning treats the enhanced samples of the same image as positive samples and all other samples as negative samples. The feature distribution of the data is learned by reducing the distance between positive sample pairs and increasing the distance between negative sample pairs.

[0003] Existing image recognition methods have limitations when dealing with negative samples. Specifically, there is a difference between the definition of negative samples in contrastive learning image recognition methods and the definition of negative samples in downstream tasks. Taking the puppy sample as an example, except for the puppy corresponding to the anchor sample, all other samples in the data space are regarded as negative samples of the anchor sample, that is, this image recognition method regards each sample as a class. However, in downstream tasks, negative samples should be samples of puppies of different breeds. Therefore, these puppy samples that are regarded as negative samples in contrastive learning are called false negative samples of image recognition. Pushing false negative samples away from the anchor sample incorrectly will cause a deviation in the semantic information in contrastive learning and downstream tasks, thereby causing errors in the classification results in downstream tasks, and thus making image recognition inaccurate.

[0004] In order to solve the problem of semantic loss caused by false negative samples in contrastive learning image recognition, some contrastive learning methods use clustering technology for image recognition. During training, the feature vectors corresponding to the samples are clustered, and the cluster labels are used as pseudo labels of the samples to guide contrastive learning. That is, the positive samples of contrastive learning are those with the same pseudo labels, and the negative samples are those with different pseudo labels. However, applying this clustering method to image recognition requires the use of the entire training set as input, which requires additional training time and space. As can be seen from the above, although the contrastive learning method of image recognition can improve the accuracy of image recognition to a certain extent, it requires more resources and time, and the results obtained by only contrastive learning for false negative samples are still difficult to distinguish similar samples, resulting in insufficient recognition accuracy of image recognition. The above problems need to be solved. Summary of the invention

[0005] In order to accurately distinguish similar samples, reduce the use of resources and time during learning, and improve the accuracy of image recognition, this application provides an image recognition method, system, device and medium for prototype weighted contrast learning, which adopts the following technical solutions: In a first aspect, the present application provides an image recognition method using prototype weighted contrast learning, comprising: Acquire sample feature information, cluster the sample feature information, and obtain first clustering information; Constructing a clustering consistency loss function based on the first clustering information of the two sample feature information to obtain a clustering result; The clustering consistency loss function is optimized, and when the clustering result is judged to be credible, the negative sample feature information is obtained, and the negative sample feature information is clustered to obtain the second clustering information; Constructing a weighted contrast learning loss function according to the first clustering information and the second clustering information to obtain a contrast result; The total loss function is constructed based on the clustering results and the comparison results to obtain the total loss information used for gradient back propagation to update the model parameters.

[0006] Preferably, the specific step of clustering the sample feature information to obtain the first clustering information includes: The sample feature information includes a number of sample feature vectors, and the number of sample feature vectors are clustered into a preset number of categories to obtain cluster prototype information containing a preset number of categories.

[0007] Preferably, the specific step of clustering the sample feature information to obtain the first clustering information further includes: Probabilistic analysis is performed on each sample feature vector, and a probability function of the category to which the sample feature vector belongs is constructed to obtain probability information.

[0008] Preferably, it also includes: The two sample feature information for constructing the clustering consistency loss function come from the same original data. Both sample feature information are subjected to probability analysis. According to the probability function corresponding to the two sample feature information, the clustering consistency loss function is constructed.

[0009] Preferably, the specific steps of optimizing the clustering consistency loss function are: Constructing a first cluster label according to the cluster prototype information, and performing maximization processing on the first cluster label; The neural network output probability similarity information is obtained, and the optimal clustering label for optimizing the clustering consistency loss function is constructed according to the maximized first clustering label and the neural network output probability similarity information.

[0010] Preferably, it also includes: An equal division constraint function is constructed for the clustering consistency loss function, and the equal division constraint function is used to divide the sample feature information equally into several clusters.

[0011] Preferably, the specific steps of obtaining sample characteristic information are: Acquire sample information, and perform enhancement processing on the sample information to obtain enhanced sample information; Regularization is performed on the enhanced sample information to obtain sample feature information.

[0012] Preferably, the specific steps of constructing a weighted contrastive learning loss function according to the first clustering information and the second clustering information are: Analyze the similarity between the first cluster label and the second cluster label to obtain similarity information, and obtain a weight coefficient for emphasizing difficult negative samples based on the similarity information; A weighted contrastive learning loss function is constructed according to the sample feature vector, the negative sample feature vector, the first cluster label, the second cluster label and the weight coefficient.

[0013] Preferably, it also includes: It is determined whether the first cluster label and the second cluster label are the same. If the first cluster label and the second cluster label are different, a weighted contrast learning loss function is constructed according to the first cluster information and the second cluster information.

[0014] In a second aspect, the present application provides an image recognition system for prototype weighted contrast learning, comprising: A first acquisition module, used to acquire sample feature information, cluster the sample feature information, and obtain first clustering information; A clustering module, used to construct a clustering consistency loss function according to the first clustering information of the two sample feature information to obtain a clustering result; A second acquisition module is used to optimize the clustering consistency loss function, and when the clustering result is judged to be credible, obtain negative sample feature information, cluster the negative sample feature information, and obtain second clustering information; A comparison module, used to construct a weighted comparison learning loss function according to the first clustering information and the second clustering information to obtain a comparison result; The total processing module is used to construct a total loss function based on the clustering results and the comparison results, and obtain the total loss information used for gradient back propagation to update the model parameters.

[0015] Preferably, the first acquisition module is used to cluster the sample feature information, and the specific steps of obtaining the first clustering information include: The sample feature information includes a number of sample feature vectors, and the number of sample feature vectors are clustered into a preset number of categories to obtain cluster prototype information containing a preset number of categories.

[0016] Preferably, the first acquisition module is used to cluster the sample feature information, and the specific steps of obtaining the first clustering information further include: Probabilistic analysis is performed on each sample feature vector, and a probability function of the category to which the sample feature vector belongs is constructed to obtain probability information.

[0017] Preferably, it also includes: The clustering module is used to construct two sample feature information of the clustering consistency loss function from the same original data, and the two sample feature information are subjected to probability analysis. The clustering consistency loss function is constructed according to the probability function corresponding to the two sample feature information.

[0018] Preferably, the specific steps of the second acquisition module for optimizing the clustering consistency loss function are: Constructing a first cluster label according to cluster prototype information, and performing maximization processing on the first cluster label; The neural network output probability similarity information is obtained, and the optimal clustering label for optimizing the clustering consistency loss function is constructed according to the maximized first clustering label and the neural network output probability similarity information.

[0019] Preferably, it also includes: The clustering module is used to construct an equal division constraint function for the clustering consistency loss function, and the equal division constraint function is used to divide the sample feature information equally into several clusters.

[0020] Preferably, the specific steps of the first acquisition module for acquiring sample feature information are: Acquire sample information, and perform enhancement processing on the sample information to obtain enhanced sample information; Regularization is performed on the enhanced sample information to obtain sample feature information.

[0021] Preferably, the specific steps of the comparison module for constructing a weighted comparison learning loss function according to the first clustering information and the second clustering information are: Analyze the similarity between the first cluster label and the second cluster label to obtain similarity information, and obtain a weight coefficient for emphasizing difficult negative samples based on the similarity information; A weighted contrastive learning loss function is constructed according to the sample feature vector, the negative sample feature vector, the first cluster label, the second cluster label and the weight coefficient.

[0022] Preferably, it also includes: The comparison module is used to determine whether the first cluster label and the second cluster label are the same. When the first cluster label and the second cluster label are different, a weighted comparison learning loss function is constructed according to the first cluster information and the second cluster information.

[0023] In a third aspect, the present application provides an image recognition device of prototype weighted contrastive learning, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the image recognition method of prototype weighted contrastive learning as described above.

[0024] In a fourth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the image recognition method of prototype weighted contrast learning as described above when running.

[0025] In summary, compared with the prior art, the technical solution provided by this application has at least the following beneficial effects: The present application obtains sample feature information, performs online clustering on the sample feature information, obtains first clustering information for constructing a loss function, and then constructs a clustering consistency loss function based on two sample feature information from the same original data, thereby obtaining a clustering result, eliminating the adverse effects of false negative samples, and optimizing the clustering consistency loss function. When the clustering result is judged to be credible, the negative sample feature information is obtained for clustering to obtain second clustering information. A weighted contrast learning loss function is constructed using the first clustering information and the second clustering information. The difficult negative sample is determined based on the distance between the negative sample and the cluster prototype corresponding to the anchor point, and a larger weight is assigned to the more difficult negative sample to obtain a contrast result, which effectively eliminates false negative samples, thereby accurately distinguishing similar samples, reducing the use of resources and time during learning, and improving the accuracy of image recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1It is a flowchart of an image recognition method using prototype weighted contrast learning described in an embodiment of the present application.

[0027] Figure 2 It is a flowchart of the overall architecture and training process of PWCL described in the embodiment of the present application.

[0028] Figure 3 It is a module diagram of an image recognition system using prototype weighted contrast learning described in an embodiment of the present application.

[0029] Description of reference numerals: 1. First acquisition module; 2. Clustering module; 3. Second acquisition module; 4. Comparison module; 5. Overall processing module. DETAILED DESCRIPTION

[0030] The following combination Figure 1-Figure 3 The present application is described in further detail. The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to be limiting.

[0031] Although existing clustering contrastive learning methods have alleviated the semantic loss problem in contrastive learning to a certain extent, these methods do not take into account the impact of hard negative samples in contrastive learning. For example, taking puppy samples as an example, hard negative samples refer to samples that are very similar to puppies but are not puppies, such as wolves, or puppy samples that are of different breeds from the anchor puppy. They have a high similarity with the anchor puppy and should be given special attention during training. Such samples are called hard negative samples because they have rich semantic information and gradient information. Pushing away from these hard negative samples helps the model learn a more uniform feature distribution, thereby improving the generalization performance of the model, the accuracy of image recognition, and facilitating the migration of the model to downstream tasks.

[0032] At present, there are a few studies that comprehensively consider false negative samples and hard negative samples. Therefore, it can be seen that the existing technology still lacks attention to false negative samples and hard negative samples. Based on the above discussion, this application addresses the problem of lack of comprehensive attention to false negative samples and hard negative samples in contrastive learning, and proposes a prototype weighted contrastive learning image recognition method Prototypical Weighted Contrastive Learning (PWCL), which emphasizes the importance of hard negative samples while excluding false negative samples.

[0033] Reference Figure 1 and 2 The present application relates to a prototype weighted contrast learning image recognition method, which specifically includes: Step S1: Acquire sample feature information, cluster the sample feature information, and obtain first clustering information; Step S2: constructing a clustering consistency loss function according to the first clustering information of the two sample feature information to obtain a clustering result; Step S3: Optimizing the clustering consistency loss function, and when the clustering result is judged to be credible, obtaining negative sample feature information, clustering the negative sample feature information, and obtaining second clustering information; Step S4: constructing a weighted contrast learning loss function according to the first clustering information and the second clustering information to obtain a contrast result; Step S5: Construct a total loss function based on the clustering results and the comparison results to obtain the total loss information used for gradient back propagation to update the model parameters.

[0034] Specifically, the main process of this application is divided into online clustering, weighted learning comparison and experimental verification. First, after obtaining the sample, the sample is enhanced and the input batch size is , Encoder , Projection head , Temperature coefficient , number of cluster categories , balance coefficient , discrete coefficient 𝜖 and other preset information, and then extract the sample feature information through the neural network encoder. The system obtains the sample feature information, performs online clustering on the sample feature information, and obtains the first clustering information used to construct the loss function, thereby improving the consistency of the clustering labels of the two enhanced samples, thereby clustering semantically similar samples into the same cluster cluster. Then, a clustering consistency loss function is constructed based on the sample feature information from the same original data to obtain the clustering result, eliminate the adverse effects of false negative samples, and optimize the clustering consistency loss function. When the clustering result is judged to be credible, the negative sample feature information is obtained for clustering to obtain the second clustering information. A weighted contrast learning loss function is constructed through the first clustering information and the second clustering information, and negative samples are selected from different clusters for contrast learning. The difficult negative samples are determined according to the distance between the negative samples and the cluster prototypes corresponding to the anchor points, and a larger weight is assigned to the more difficult negative samples to obtain the contrast result. This application optimizes both clustering loss and weighted contrast loss during the training process, effectively eliminating false negative samples, thereby accurately distinguishing similar samples, reducing the use of resources and time during learning, improving the generalization ability of the contrast model, and improving the accuracy of image recognition.

[0035] As one implementation manner, the specific steps of clustering the sample feature information to obtain the first clustering information include: The sample feature information includes a number of sample feature vectors, and the number of sample feature vectors are clustered into a preset number of categories to obtain cluster prototype information containing a preset number of categories.

[0036] Specifically, for each enhanced sample , first use the neural network encoder and projection head Get the feature vector of the sample , is the dimension of the vector.

[0037] A feature vector is a representation of a data point in a feature space, usually consisting of multiple features. Each feature vector can be regarded as a data point, whose position in the multidimensional space is determined by its feature value, that is, by its feature.

[0038] The embodiment of the present application clusters all feature vectors in the same batch. Clustering is an unsupervised learning technique that groups or aggregates samples in a data set into multiple categories, making samples in the same category more similar and samples in different categories more different. Feature vectors are used to define data points, and the clustering algorithm assigns them to different categories based on the similarity between feature vectors.

[0039] Specifically, it is clustered into categories, each category's cluster prototype, that is, the cluster center of each category consists of a prototype feature vector Indicates that there are Cluster prototypes: The cluster prototype feature vector of each category can be understood as the center or representative of the category, usually the average value or center point of all feature vectors in the category, representing the characteristic characteristics of the category. Used to indicate Cluster prototypes, Each column is a cluster prototype .

[0040] As one implementation method, the specific steps of obtaining sample feature information are: Acquire sample information, and perform enhancement processing on the sample information to obtain enhanced sample information; Regularization is performed on the enhanced sample information to obtain sample feature information.

[0041] Specifically, using a neural network encoder and projection head Get the feature vector of the sample , is the dimension of the vector, and then the eigenvector is Regularization: .

[0042] Eigenvector regularization refers to adjusting or constraining eigenvectors in machine learning to improve the generalization and performance of the model. Specifically, regularization is achieved through different methods, the main purpose of which is to prevent the model from overfitting the training data, thereby improving the performance of the model on unseen data.

[0043] Regularization reduces the overfitting of the model on the training data by limiting the complexity of the model, making it simpler and more general. When the model is too complex, it may remember the details and noise of the training data and fail to generalize to new data. Through regularization, the adjustment of the feature vector enables the model to better capture the general characteristics of the data, rather than overfitting specific training samples, which can improve the model's prediction accuracy and stability for new data.

[0044] As one implementation manner, the specific step of clustering the sample feature information to obtain the first clustering information further includes: Probabilistic analysis is performed on each sample feature vector, and a probability function of the category to which the sample feature vector belongs is constructed to obtain probability information.

[0045] Specifically, the probability analysis performed in the embodiment of the present application can obtain the result of the association degree between each data point and each category, rather than simply rigidly classifying it into a certain category. , calculate that it belongs to The probability of the class is: in, is the temperature coefficient.

[0046] And for another enhanced sample from the same original data, the corresponding feature vector , using the same method to calculate it belongs to The probability of the class is: As one of the implementation methods, two sample feature information for constructing the clustering consistency loss function come from the same original data, and probability analysis is performed on both sample feature information. A clustering consistency loss function is constructed based on the probability function corresponding to the two sample feature information.

[0047] Specifically, because the two enhanced samples and They come from the same original data, so the two should be able to predict each other's clustering results, that is, the feature information of the two samples maintains the consistency of the clustering labels. The average cross entropy loss function is used to define the goal of label consistency: Among them, 𝑞(𝑦∣𝒛𝑖′) is the clustering label corresponding to sample 𝒙𝑖′, and 𝑞(𝑦∣𝒛𝑖′) is regarded as the true label of sample 𝒙𝑖. Similarly, the clustering result of sample 𝒙𝑖′ is predicted based on the clustering label of sample 𝒙𝑖: Therefore, the cluster consistency loss is expressed as: As one implementation method, the specific steps for optimizing the clustering consistency loss function are: Constructing a first cluster label according to cluster prototype information, and performing maximization processing on the first cluster label; The neural network output probability similarity information is obtained, and the optimal clustering label for optimizing the clustering consistency loss function is constructed according to the maximized first clustering label and the neural network output probability similarity information.

[0048] Specifically, for a batch of samples of size 𝑁, the feature vector , , are assigned to 𝐾 cluster prototypes , . Assume that the cluster label of the sample is , , then the best clustering label 𝑸 can be solved by maximizing the similarity between 𝑸 and the sample classification probability output by the neural network. That is: in: 𝐻(𝑸) is used to calculate the entropy of cluster label 𝑸, and 𝜖 is used to control the discreteness of label distribution. Then, in order to make each cluster obtain evenly distributed samples, the idea of ​​optimal transportation is used to restrict 𝑸, and we get: in, is a 𝐾-dimensional vector of all ones, is a 𝑁-dimensional all-1 vector. This constraint ensures that at least The Sinkhorn-Knopp algorithm is used to solve the optimal transportation problem and obtain the optimal clustering label. : in, and are two renormalized vectors, which are solved by iterative matrix multiplication. 𝜖 is used to balance the convergence speed of the model and the discreteness of the clustering labels.

[0049] Through the above calculation, the optimal clustering label is obtained The optimal cluster labels are calculated using the Sinkhorn-Knopp algorithm. , which is used to calculate the result after considering the equal division constraint and maximizing the similarity between the cluster labels and the neural network output. is a continuous solution, that is, the probability distribution of each sample belonging to each cluster, rather than a hard label. Then, the optimal cluster label is used to optimize the cluster consistency loss function in the above formula. The specific steps are as follows: First, calculate the cross entropy loss. Use the optimal cluster label Calculate the cross entropy loss for each sample, that is, calculate the probability formula of two enhanced samples of the same original data.

[0050] The next step is cluster label consistency. It is the optimal label after considering the equal division constraint, which can effectively avoid trivial solutions and ensure that the samples in each batch are evenly distributed to each cluster.

[0051] Finally, merge the losses. Merge the cross entropy losses of all samples to form the final clustering consistency loss , which is the above clustering consistency loss function.

[0052] As one implementation method, the present application proposes a multi-cropping strategy, which uses two random crops of the original size for the original image, and additional random crops smaller than the original size, thereby improving the robustness of contrastive learning. At this time, the cluster consistency loss can be expressed as: At this time, a total of 𝑉+2 enhanced samples are used for cluster consistency calculation, but only 2 enhanced samples cropped to the original size are still used for cluster label calculation. Therefore, enhanced samples usually increase the diversity of training data by transforming or expanding the original data, which helps to verify the stability and robustness of clustering results after data changes.

[0053] By comparing the clustering labels of the two enhanced samples cropped at the original size, the impact of data enhancement on the clustering results can be evaluated. If the clustering results of the two samples are consistent, it means that data enhancement has retained the structural information of the data to some extent, which helps to improve the generalization ability of the model. Cluster consistency calculation can be used to verify the consistency of the clustering algorithm under different input data transformations. By comparing the clustering results of different enhanced samples, the stability and reliability of the algorithm when processing enhanced data can be evaluated.

[0054] Using V+2 augmented samples can increase the diversity of the data, thereby more comprehensively evaluating the performance of the clustering model. This approach helps to discover the strengths and weaknesses of the model under various data transformations and guide further model improvements or adjustments.

[0055] In practical applications, models need to be able to handle diverse data, and data augmentation can help models better generalize to unseen data. Through the above methods, we can verify the specific impact of data augmentation on the generalization ability of the model, thereby optimizing the model design and data augmentation strategy.

[0056] As one implementation mode, it also includes: An equal division constraint function is constructed for the clustering consistency loss function, and the equal division constraint function is used to divide the sample feature information equally into several clusters.

[0057] Specifically, the cluster consistency loss function of the minimization formula can be optimized by dividing all samples of the same batch into the same cluster cluster, which may cause the model to fall into a trivial solution. In order to reduce this phenomenon, an equal division constraint is added to the loss function, that is, a batch of samples should be equally divided into 𝐾 clusters.

[0058] In machine learning models, an equal-division constraint is added to the loss function, that is, in the optimization process, not only the minimization of the loss function itself is considered, but also certain equal-division conditions or constraints must be met. This constraint usually involves balancing the performance of the loss function on different samples so that the prediction error or loss of the model on each sample will not have too large deviations, thereby improving the overall performance and robustness of the model. The purpose of the constraint is to ensure that each cluster has an appropriate number of samples in each batch, thereby promoting the model to learn a more stable and representative clustering pattern.

[0059] As one implementation mode, it also includes: It is determined whether the first cluster label and the second cluster label are the same. If the first cluster label and the second cluster label are different, a weighted contrast learning loss function is constructed according to the first cluster information and the second cluster information.

[0060] Specifically, after obtaining the clustering label of each sample, the embodiment of the present application will use negative samples extracted from different clusters for comparative learning, thereby eliminating the influence of false negative samples to a certain extent. In the initial stage of training, the clustering results of the model are not stable enough, and the generated clustering pseudo-labels are still unreliable in representing the semantic relationship between samples. Therefore, the method proposed in this application is only used in the later stage of training. The optimal clustering label is calculated using the Sinkhorn-Knopp algorithm. When , the cluster labels gradually become reliable, and then the contrastive learning loss is introduced. Given an anchor sample, its corresponding negative sample should have a different cluster label from it, so that the sampling strategy can make the semantic difference between the anchor sample and the negative sample, reducing the adverse effect of false negative samples. At this time, the formula in: The contrastive learning loss function in can be modified as: in: in, and is the feature vector of the two samples and The corresponding cluster labels are is an indicator function, and a comparative study is performed on the two samples if and only if their cluster labels are different.

[0061] In this way, negative samples that belong to the same cluster as the anchor samples, i.e., false negative samples, can be excluded.

[0062] As one implementation method, the specific steps of constructing a weighted contrastive learning loss function according to the first clustering information and the second clustering information are: Analyze the similarity between the first cluster label and the second cluster label to obtain similarity information, and obtain a weight coefficient for emphasizing difficult negative samples based on the similarity information; A weighted contrastive learning loss function is constructed according to the sample feature vector, the negative sample feature vector, the first cluster label, the second cluster label and the weight coefficient.

[0063] Specifically, in addition to selecting negative samples according to the cluster to which the samples belong and excluding false negative samples, the present embodiment of the application selects the negative samples according to the prototype. and The similarity formula between them calculates the distance between the cluster prototypes corresponding to the samples and obtains the weight coefficient The formula is defined to assign different weights to different negative samples. When the distance between the prototype corresponding to the negative sample and the prototype corresponding to the anchor point is small, these negative samples should have more similar semantic information to the anchor point samples, so paying attention to these negative samples helps the model learn more useful information. On the contrary, when the distance between the prototypes corresponding to the two is too large, these negative samples and anchor points can be easily distinguished from the anchor point samples. If too much attention is paid to these simple negative samples, the model will not learn useful knowledge. Therefore, the weight, that is, the importance, of each negative sample can be determined according to the distance between the prototypes. At this time, the contrastive learning loss function is modified as follows: in, Still represents the cosine similarity of the feature vectors corresponding to the two samples, is the cluster label corresponding to the sample, It is a negative sample Relative to the anchor sample The weight of is a regularization coefficient.

[0064] Prototype and The similarity between them is: Then the weight coefficient in the loss function formula is is defined as: in, , The above normalization operation converts the weight coefficient The value of is limited to [0,1] and is proportional to the similarity of the prototype, that is, the more similar the prototype is, the larger the weight coefficient corresponding to the negative sample is, thus emphasizing the difficult negative samples.

[0065] Weighted contrastive learning assigns larger weights to negative samples whose corresponding prototypes are closer to the prototypes of anchor samples, and smaller weights to negative samples whose prototypes are farther away.

[0066] In summary, the final loss function of the image recognition method of prototype weighted contrast learning proposed in this application is: Among them, 𝜆 is used to balance the clustering consistency loss and the weighted contrast loss. This loss function optimizes the clustering prototype 𝑪 and the neural network encoder simultaneously.

[0067] The calculation process of the image recognition method of prototype weighted contrast learning in the embodiment of the present application is as follows: 1. Initialization parameters: input batch size 𝑁, neural network encoder , Projection head , temperature coefficient 𝜏, number of cluster categories 𝐾, balance coefficient 𝜆, and dispersion coefficient 𝜖.

[0068] 2. Model training loop: loop when the model has not converged.

[0069] 3. Batch sample processing: For each batch of samples , that is, for the original sample: Get enhanced samples and .

[0070] The enhanced sample is input into the neural network encoder and projection head to obtain the feature vector and .

[0071] 4. Calculate cluster labels: Calculate cluster labels and , and calculate the sample based on the eigenvector The cluster labels .

[0072] 5. Calculate category probability: Calculate the category probability of the network output and .

[0073] 6. Get the clustering prototype: Get the parameters of the last layer of the network as the clustering prototype .

[0074] 7. Negative sample processing, for each sample : Calculation Sample The cluster labels .

[0075] If 𝑗∉{𝑖,𝑖′} and ≠ , then calculate the weight of negative samples , and calculate the weighted contrast loss.

[0076] 8. Calculate loss: Calculate cluster consistency loss and calculate the total loss.

[0077] 9. Update model parameters: Update model parameters through gradient back propagation.

[0078] 10. End of loop: After the model converges, return to the neural network encoder.

[0079] This application first selects five benchmark image datasets in the field of contrastive learning, including four small and medium-sized datasets, CIFAR-10, STL-10, CIFAR-100 and SVHN, and a large-size dataset ImageNet-100. Based on the original image data samples, the data augmentation operation is performed, and the multi-cropping strategy is proposed. For the original image samples, two random crops of the original size are used, and additional random crops smaller than the original size are used to construct enhanced samples. Specifically, for small and medium-sized datasets, the pictures generated by the multi-cropping strategy are 2 pictures of the original size and 4 pictures of the original size / 4. For ImageNet-100, the pictures generated by the multi-cropping strategy are 2 pictures of size 224 and 6 pictures of size 96.

[0080] The enhanced sample data of the above data are divided into training set and test set. ResNet-50 is used as the neural network encoder to extract the features of the enhanced sample data, and the output feature dimension is 2048. Specifically, for the four small and medium-sized data sets, the same operation as SimCLR is used to replace the first 7×7 convolution with a stride of 2 of ResNet-50 with a 3×3 convolution with a stride of 1, and the first maximum pooling operation is removed. For ImageNet-100, the original structure of ResNet-50 is maintained. During the training phase, the output of the encoder is input into a multi-layer perceptron projection head with one hidden layer. The feature dimension of the hidden layer is 2048, and the final feature dimension of the projection head output is 128.

[0081] In order to propose an online clustering method, each batch of training samples is clustered. This method uses the number of cluster categories 𝐾 as the number of neurons output by the last layer of the neural network, and the cluster prototype to be learned as the weight of the first hidden layer of the neural network. As the training progresses, the cluster prototype is continuously updated, and the probability value of the current sample belonging to each cluster category is output.

[0082] Prototype weighted contrastive learning for image recognition is proposed. After obtaining the cluster label of each sample, this method will use negative samples extracted from different clusters for contrastive learning, thereby eliminating the influence of false negative samples to a certain extent. And this method assigns different weights to different negative samples according to the distance between the cluster prototypes corresponding to the samples. That is, a larger weight is assigned to those negative samples whose corresponding prototypes are closer to the prototypes of the anchor samples, while a smaller weight is assigned to those negative samples whose prototypes are farther apart. The more similar the prototypes are, the larger the weight coefficient corresponding to the negative samples is, thereby emphasizing difficult negative samples.

[0083] The method is also compared with several most representative methods in the field of contrastive learning to verify its effectiveness.

[0084] This application experimentally verifies the proposed image recognition method of prototype weighted contrastive learning in three fields, namely image representation learning, and compares it with several most representative methods in the field of contrastive learning to verify the effectiveness of the method. The experimental steps are as follows: This application selected five benchmark image datasets in the field of contrastive learning for experiments, including: 4 small and medium-sized datasets, including CIFAR-10, STL-10, CIFAR-100 and SVHN, and a large-size dataset ImageNet-100.

[0085] First, the model is trained using the training set in the dataset, and then the model parameters are fixed. The output of the model is used as the input of a linear classifier to perform classification prediction on the benchmark dataset, and the classification results are used as the evaluation criteria for the method performance.

[0086] Among them, this application compares the proposed prototype weighted contrastive learning image recognition method with several representative methods in the field of contrastive learning. The contrast methods used are: (1) Classic method, SimCLR: InfoNCE loss is applied to contrastive learning to perform instance-level contrastive learning. Each sample is regarded as a class. MoCov2: Based on MoCo, a projection head is added and more data enhancement is used. Queues are used as memory banks to store negative samples. BYOL: An asynchronous asymmetric network architecture is used to use only positive samples for contrastive learning. (2) Methods that focus on hard negative samples, Un-Mix: Multiple mixing strategies are used on images to synthesize hard negative samples for contrastive learning. (3) Methods that eliminate false negative samples. SwAV: By maintaining the consistency of the clustering labels of positive samples, the adverse effects of false negative samples on semantics are eliminated. (4) Methods that focus on both hard negative samples and false negative samples, HCL: Debias the probability distribution of negative samples and give hard negative samples a greater probability. WCL: Determine the difficulty of negative samples based on the difference between them on biased and unbiased encoders, and debias the probability distribution of negative samples.

[0087] Among them, this application uses a linear classifier to classify and predict the feature vector obtained from the encoder. The linear classifier is trained using the training set, and the classification accuracy is calculated using the test set. The experiment was repeated 10 times on each data set, and the classification accuracy of the image data set in Table 1 (unit: %) was obtained: Table 1 Classification accuracy of image datasets As can be seen from the table, the method proposed in this application achieves the highest classification accuracy in most cases, proving that this method can preserve the semantic information of the data and effectively weaken the adverse effects of false negative samples.

[0088] Compared with SwAV, a method that only uses clustering to retain semantic information, the learning effect of this application has been further improved, proving that giving greater weight to difficult negative samples can make the model focus on more difficult negative samples, allowing the model to learn more meaningful representations, thereby improving model performance.

[0089] Compared with two methods (HCL and WCL) that focus on both false negative samples and hard negative samples, this method further improves the classification accuracy, especially on large-size datasets. The performance of the method has been greatly improved, thanks to the fact that online clustering can retain as much semantic information as possible based on the model performance, rather than using prior knowledge to retain semantic information.

[0090] Moreover, MoCov2 and Un-Mix achieved higher classification accuracy on ImageNet-100, thanks to their use of memory banks and queues to store a large number of negative samples.

[0091] In addition, during the training of the encoder and projection head, this method uses a memory bank to store some labeled samples, selects the first 200 nearest neighbor samples, calculates the KNN classification accuracy for the training samples, and records the highest calculation result. The experimental results are shown in Table 2: Table 2 KNN classification accuracy of image dataset (unit: %) The changing trends of KNN classification accuracy and linear classification accuracy are roughly the same. The method proposed in this paper can obtain a higher KNN classification accuracy during training, which proves that this method can effectively retain the semantic information of samples.

[0092] In particular, for CIFAR-10, although the method proposed in this paper did not achieve the highest KNN accuracy, it was still able to achieve the highest linear classification accuracy during the linear classification process.

[0093] Combining the two classification results, it can be concluded that the method proposed in this application effectively avoids the semantic loss problem caused by false negative samples, and further improves the effect of the model by emphasizing hard negative samples.

[0094] The method proposed in this application uses clustering technology to retain semantic information, so this application calculates the clustering quality of the model. The calculation results are shown in Table 3. This experiment mainly compares the contrastive learning method using clustering and the classic contrastive learning method.

[0095] Table 3 Clustering quality on CIFAR-10 (unit: %) As can be seen from the table, compared with the instance-level contrastive learning method SimCLR, the method proposed in this application uses clustering technology to effectively retain the semantic information of the sample, thereby producing a better clustering effect and significantly improving the clustering index.

[0096] Compared with the clustering method SwAV, the method proposed in this application emphasizes hard negative samples, enabling the model to learn more effective representations, thereby further improving the clustering effect.

[0097] In summary, through experimental verification in the image field, it can be seen that the method proposed in this application can retain the semantic information of the data through clustering, and emphasize the important role of hard negative samples through weighting, thereby effectively improving the performance of the existing model.

[0098] Reference Figure 3 , an image recognition system based on prototype weighted contrast learning is provided for an embodiment of the present application, the system comprising a first acquisition module, a clustering module, a second acquisition module, a contrast module and a total processing module.

[0099] A first acquisition module, used to acquire sample feature information, cluster the sample feature information, and obtain first clustering information; A clustering module, used to construct a clustering consistency loss function according to the first clustering information of the two sample feature information to obtain a clustering result; A second acquisition module is used to optimize the clustering consistency loss function, and when the clustering result is judged to be credible, obtain negative sample feature information, cluster the negative sample feature information, and obtain second clustering information; A comparison module, used to construct a weighted comparison learning loss function according to the first clustering information and the second clustering information to obtain a comparison result; The total processing module is used to construct a total loss function based on the clustering results and the comparison results, and obtain the total loss information used for gradient back propagation to update the model parameters.

[0100] As one implementation manner, the first acquisition module is used to cluster the sample feature information, and the specific steps of obtaining the first clustering information include: The sample feature information includes a number of sample feature vectors, and the number of sample feature vectors are clustered into a preset number of categories to obtain cluster prototype information containing a preset number of categories.

[0101] As one implementation manner, the first acquisition module is used to cluster the sample feature information, and the specific steps of obtaining the first clustering information further include: Probabilistic analysis is performed on each sample feature vector, and a probability function of the category to which the sample feature vector belongs is constructed to obtain probability information.

[0102] As one implementation mode, it also includes: The clustering module is used to construct two sample feature information of the clustering consistency loss function from the same original data, and the two sample feature information are subjected to probability analysis. The clustering consistency loss function is constructed according to the probability function corresponding to the two sample feature information.

[0103] As one implementation method, the specific steps of the second acquisition module for optimizing the clustering consistency loss function are: Constructing a first cluster label according to the cluster prototype information, and performing maximization processing on the first cluster label; The neural network output probability similarity information is obtained, and the optimal clustering label for optimizing the clustering consistency loss function is constructed according to the maximized first clustering label and the neural network output probability similarity information.

[0104] As one implementation mode, it also includes: The clustering module is used to construct an equal division constraint function for the clustering consistency loss function, and the equal division constraint function is used to divide the sample feature information equally into several clusters.

[0105] As one implementation manner, the specific steps for the first acquisition module to acquire sample feature information are: Acquire sample information, and perform enhancement processing on the sample information to obtain enhanced sample information; Regularization is performed on the enhanced sample information to obtain sample feature information.

[0106] As one implementation manner, the specific steps of the comparison module for constructing a weighted comparison learning loss function according to the first clustering information and the second clustering information are as follows: Analyze the similarity between the first cluster label and the second cluster label to obtain similarity information, and obtain a weight coefficient for emphasizing difficult negative samples based on the similarity information; A weighted contrastive learning loss function is constructed according to the sample feature vector, the negative sample feature vector, the first cluster label, the second cluster label and the weight coefficient.

[0107] As one implementation mode, it also includes: The comparison module is used to determine whether the first cluster label and the second cluster label are the same. When the first cluster label and the second cluster label are different, a weighted comparison learning loss function is constructed according to the first cluster information and the second cluster information.

[0108] An embodiment of the present application provides an image recognition device using prototype weighted contrastive learning, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the image recognition method using prototype weighted contrastive learning as described above.

[0109] An embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the image recognition method of prototype weighted contrast learning as described above when running.

[0110] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and product can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0111] In several embodiments provided in this application, it should be understood that the disclosed methods, systems, devices and program products may be implemented in other ways.

[0112] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0113] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A prototype weighted contrast learning image recognition method, characterized in that: include: Acquire sample feature information, cluster the sample feature information, and obtain first clustering information; Constructing a clustering consistency loss function based on the first clustering information of the two sample feature information to obtain a clustering result; The clustering consistency loss function is optimized, and when the clustering result is judged to be credible, the negative sample feature information is obtained, and the negative sample feature information is clustered to obtain the second clustering information; Constructing a weighted contrast learning loss function according to the first clustering information and the second clustering information to obtain a contrast result; The total loss function is constructed based on the clustering results and the comparison results to obtain the total loss information used for gradient back propagation to update the model parameters.

2. The image recognition method of prototype weighted contrast learning according to claim 1, characterized in that: The specific steps of clustering the sample feature information to obtain the first clustering information include: The sample feature information includes a number of sample feature vectors, and the number of sample feature vectors are clustered into a preset number of categories to obtain cluster prototype information containing a preset number of categories.

3. The image recognition method of prototype weighted contrast learning according to claim 2, characterized in that: The specific step of clustering the sample feature information to obtain the first clustering information further includes: Probabilistic analysis is performed on each sample feature vector, and a probability function of the category to which the sample feature vector belongs is constructed to obtain probability information.

4. The image recognition method of prototype weighted contrast learning according to claim 3, characterized in that: Also includes: The two sample feature information for constructing the clustering consistency loss function come from the same original data. Both sample feature information are subjected to probability analysis. According to the probability function corresponding to the two sample feature information, the clustering consistency loss function is constructed.

5. The image recognition method of prototype weighted contrast learning according to claim 3, characterized in that: The specific steps of optimizing the clustering consistency loss function are: Constructing a first cluster label according to cluster prototype information, and performing maximization processing on the first cluster label; The neural network output probability similarity information is obtained, and the optimal clustering label for optimizing the clustering consistency loss function is constructed according to the maximized first clustering label and the neural network output probability similarity information.

6. The image recognition method of prototype weighted contrast learning according to claim 1, characterized in that: Also includes: An equal division constraint function is constructed for the clustering consistency loss function, and the equal division constraint function is used to divide the sample feature information equally into several clusters.

7. The image recognition method of prototype weighted contrast learning according to claim 1, characterized in that: The specific steps of obtaining sample feature information are: Acquire sample information, and perform enhancement processing on the sample information to obtain enhanced sample information; Regularization is performed on the enhanced sample information to obtain sample feature information.

8. The image recognition method of prototype weighted contrast learning according to claim 5, characterized in that: The specific steps of constructing the weighted contrast learning loss function according to the first clustering information and the second clustering information are: Analyze the similarity between the first cluster label and the second cluster label to obtain similarity information, and obtain a weight coefficient for emphasizing difficult negative samples based on the similarity information; A weighted contrastive learning loss function is constructed according to the sample feature vector, the negative sample feature vector, the first cluster label, the second cluster label and the weight coefficient.

9. The image recognition method of prototype weighted contrast learning according to claim 8, characterized in that: Also includes: It is determined whether the first cluster label and the second cluster label are the same. If the first cluster label and the second cluster label are different, a weighted contrast learning loss function is constructed according to the first cluster information and the second cluster information.

10. A prototype weighted contrastive learning image recognition system, characterized in that: include: A first acquisition module, used to acquire sample feature information, cluster the sample feature information, and obtain first clustering information; A clustering module, used to construct a clustering consistency loss function according to the first clustering information of the two sample feature information to obtain a clustering result; A second acquisition module is used to optimize the clustering consistency loss function, and when the clustering result is judged to be credible, obtain negative sample feature information, cluster the negative sample feature information, and obtain second clustering information; A comparison module, used to construct a weighted comparison learning loss function according to the first clustering information and the second clustering information to obtain a comparison result; The total processing module is used to construct a total loss function based on the clustering results and the comparison results, and obtain the total loss information used for gradient back propagation to update the model parameters.

11. The image recognition system of prototype weighted contrastive learning according to claim 10, characterized in that: The first acquisition module is used to cluster the sample feature information, and the specific steps of obtaining the first clustering information include: The sample feature information includes a number of sample feature vectors, and the number of sample feature vectors are clustered into a preset number of categories to obtain cluster prototype information containing a preset number of categories.

12. The image recognition system of prototype weighted contrastive learning according to claim 11, characterized in that: The first acquisition module is used to cluster the sample feature information, and the specific steps of obtaining the first clustering information also include: Probabilistic analysis is performed on each sample feature vector, and a probability function of the category to which the sample feature vector belongs is constructed to obtain probability information.

13. The image recognition system of prototype weighted contrastive learning according to claim 12, characterized in that: Also includes: The clustering module is used to construct two sample feature information of the clustering consistency loss function from the same original data, and the two sample feature information are subjected to probability analysis. The clustering consistency loss function is constructed according to the probability function corresponding to the two sample feature information.

14. The image recognition system of prototype weighted contrastive learning according to claim 12, characterized in that: The specific steps of the second acquisition module for optimizing the cluster consistency loss function are: Constructing a first cluster label according to cluster prototype information, and performing maximization processing on the first cluster label; The neural network output probability similarity information is obtained, and the optimal clustering label for optimizing the clustering consistency loss function is constructed according to the maximized first clustering label and the neural network output probability similarity information.

15. The image recognition system of prototype weighted contrastive learning according to claim 10, characterized in that: Also includes: The clustering module is used to construct an equal division constraint function for the clustering consistency loss function, and the equal division constraint function is used to divide the sample feature information equally into several clusters.

16. The image recognition system of prototype weighted contrastive learning according to claim 10, characterized in that: The specific steps of the first acquisition module for acquiring sample feature information are: Acquire sample information, and perform enhancement processing on the sample information to obtain enhanced sample information; Regularization is performed on the enhanced sample information to obtain sample feature information.

17. The image recognition system of prototype weighted contrastive learning according to claim 14, characterized in that: The specific steps of the comparison module for constructing a weighted comparison learning loss function according to the first clustering information and the second clustering information are: Analyze the similarity between the first cluster label and the second cluster label to obtain similarity information, and obtain a weight coefficient for emphasizing difficult negative samples based on the similarity information; A weighted contrastive learning loss function is constructed according to the sample feature vector, the negative sample feature vector, the first cluster label, the second cluster label and the weight coefficient.

18. The image recognition system of prototype weighted contrastive learning according to claim 17, characterized in that: Also includes: The comparison module is used to determine whether the first cluster label and the second cluster label are the same. When the first cluster label and the second cluster label are different, a weighted comparison learning loss function is constructed according to the first cluster information and the second cluster information.

19. An image recognition device using prototype weighted contrastive learning, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the image recognition method of prototype weighted contrast learning according to any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the image recognition method of prototype weighted contrast learning described in any one of claims 1 to 9 when running.

Images