Sample mining method based on multi-network self-supervised learning
Through the sample mining method based on multi-network self-supervised learning, using data augmentation and multi-network structures, the problem of low sample utilization efficiency in the existing technology is solved, and the model performance improvement and image classification accuracy improvement under limited resources are achieved.
Patent Information
- Application Number
- CN202411945678.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-27
- Publication Date
- 2025-04-25
AI Technical Summary
The prior art is difficult to effectively select and utilize pre-trained samples in self-supervised learning, resulting in limited model performance improvement, especially when computing resources are limited.
A sample mining method based on multi-network self-supervised learning is adopted to generate multiple views through data augmentation, combine online networks and target networks, and use positive sample mining modules and negative sample mining modules to calculate the loss value, and perform gradient updates to improve model performance.
Through the multi-network self-supervised learning method, model performance can be improved under limited resources, image classification accuracy, and pre-trained samples can be effectively utilized.
Smart Images
Figure CN119888403A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of self-supervised visual representation learning, and in particular to a sample mining method based on multi-network self-supervised learning. Background Art
[0002] In recent years, with the breakthrough of artificial intelligence technology, large model pre-training has shown powerful capabilities in fields such as natural language processing and computer vision. However, the success of these models depends largely on a large number of pre-training samples. Acquiring and processing such a large data set has become a major challenge, especially when ensuring data quality and diversity while coping with the huge demand for computing resources. In addition, data security and privacy protection have also put forward new requirements for large-scale data collection. Therefore, how to effectively select and use pre-training samples with limited resources to improve model performance has become one of the important issues that need to be solved urgently.
[0003] The quality and quantity of training data are crucial for model training, but obtaining large amounts of labeled data can be extremely expensive. Self-supervised learning is a promising solution that is rapidly gaining ground in various fields and is expected to become the future of machine learning.
[0004] Self-supervised learning extracts useful information from abundant unlabeled data for learning. In the field of computer vision, self-supervised learning has even surpassed many supervised learning methods in some cases. Unlike traditional supervised learning methods that rely on labeled data to complete specific tasks, self-supervised learning aims to discover more general and intrinsic representations that can help multiple tasks. Not only in computer vision, self-supervised learning has also made significant progress in natural language processing, multimodal learning, video analysis, audio processing and other fields.
[0005] Contrastive learning is one of the most successful paradigms in self-supervised learning, which trains the network by maximizing the similarity between pairs of positive samples and minimizing the similarity between pairs of negative samples. General methods use data augmentation transformations to generate new positive samples, thereby bringing diversity while maintaining basic semantics. Different versions of the same input image are considered as positive samples, in which case the input image belongs to the same category as the positive sample. There are some methods that construct positive samples through sample mining, but due to the lack of label information, it is not guaranteed whether these positive sample pairs belong to the same category. At the same time, in order to prevent the model from collapsing, other input images are usually regarded as negative samples. This method is to provide a large number of negative samples to the model by increasing the batch size or providing a memory bank. These strategies are crucial to prevent the model from overfitting to specific positive samples and promoting broader and more effective feature representations.
[0006] For typical contrastive self-supervised learning with positive and negative samples, the challenge lies in selecting and mining reasonable and appropriate potential samples for model training. Considering enhanced samples as positive samples will ignore other positive samples in the dataset, thus lacking diversity. However, sampling positive samples from mined data makes it difficult to guarantee the authenticity of the samples, that is, whether these samples really belong to the same category as the query samples. The same dilemma also exists in negative samples. Negative samples that are too difficult may be false negative samples, that is, negative samples that are difficult to distinguish from positive samples and are highly similar to positive samples. And negative samples that are too simple may be meaningless for model training, that is, uninformative negative samples can hardly provide key semantic information for model training. Summary of the invention
[0007] The purpose of the present invention is to provide a sample mining method based on multi-network self-supervised learning to solve the problems raised in the above background technology.
[0008] In order to solve the above technical problems, the present invention provides the following technical solutions:
[0009] A sample mining method based on multi-network self-supervised learning, the method comprising:
[0010] Step 1: Obtain a training set under a certain training batch, perform data enhancement on each image in the training set, and obtain one strongly enhanced image and two weakly enhanced images based on each image;
[0011] Step 2: Input the three images into the corresponding online network and target network respectively to obtain the corresponding image representations;
[0012] Step 3: Consider the strongly enhanced image and the weakly enhanced image of a certain image as positive samples, and the remaining images in the training set as negative samples, and calculate the loss value between the output results of the online network and the projection head to obtain the first loss;
[0013] Step 4: The two weakly enhanced images of a certain image are regarded as positive samples, and the remaining images in the training set and the samples in the repository are regarded as negative samples. The second loss and the third loss are calculated respectively by combining the positive sample mining module and the negative sample mining module through the online network, the target network and the projection head;
[0014] Step 5: Combine the three losses in step 3 and step 4, and perform gradient update on the parameters of the online network and the target network by minimizing the total loss value;
[0015] Step 6: Use the target network to update the sample repository. After multiple iterations of training, the pre-training is completed.
[0016] Preferably, the data enhancement in step 1 includes:
[0017] Perform data augmentation on the images in the training set;
[0018] A1. Weak data augmentation strategies include random cropping and resizing, horizontal flipping, color distortion, and grayscale conversion:
[0019] Define the set of strategies for weak data enhancement as Then for the input image x, according to the formula: Get the transformed image x i ;
[0020] A2. Strong data enhancement strategies include random enhancement and jigsaw operations:
[0021] The strategy set for strong data enhancement is defined as Then for the input image x, according to the formula: Get the transformed image x i .
[0022] Preferably, the online network and the target network in step 2 include:
[0023] The online network and the target network include two parts: an encoder and a projection head, wherein the online network is updated using a gradient, and the target network is updated by an exponential moving average with the help of a back-propagation network.
[0024] Preferably, step 3 comprises:
[0025] The strong data enhancement and weak data enhancement of the same image are regarded as a pair of positive samples, and other images are regarded as negative samples;
[0026] Assuming the batch size of training images is N, there are a total of 2N-2 images as negative samples;
[0027] For the obtained strong data enhancement image x3 and weak data enhancement image x1, through the online network and projection head, the projection features q3 and q1 are finally obtained. At this time, the first loss value is calculated according to the loss function
[0028]
[0029] Where exp represents the exponential function, represents negative samples, s represents similarity measure, here cosine similarity is used; t represents temperature hyperparameter, N represents the size of each batch, 2 ( N-1 ) It represents the number of negative samples in each batch.
[0030] Preferably, the positive sample mining module in step 4 includes: nearest neighbor mining, positive sample hard mining and positive sample soft mining;
[0031] Consider two different weak data enhancements of the same image as a pair of positive samples;
[0032] S1, the nearest neighbor mining: for each sample feature z1 of the online branch, the corresponding sample feature z2 of the target branch is used to mine the expected positive samples in the repository M, and the cosine similarity between samples is used to mine the top k most similar features from the memory library:
[0033]
[0034] where NN ( z 2) It means mining k feature samples that are most similar to z2 from the memory bank M;
[0035] S2, the positive sample hard mining: the enhanced sample view z2 is another feature representation of the same type of sample z1. For the determined positive sample z2, the hard weighting method is adopted, that is, the loss weight w0 is set to 1, and the second loss value is calculated:
[0036]
[0037] Where w0 is the weight coefficient of the loss, represents negative samples, 2 ( N-1 ) is the number of negative samples in the current batch;
[0038] S3, the positive sample soft mining: Considering that there are multiple positive samples, when calculating the contrast loss, the mined samples are not directly regarded as positive samples, but each mined positive sample NN is given ( z 2)i Assign weight w i :
[0039]
[0040] According to NN ( z 2)i The similarity ratio between z1 and the total similarity is used to measure the importance of each mined sample, where
[0041] Get the positive sample soft mining loss value, that is, the third loss value:
[0042]
[0043] where w i represents the weight coefficient of each loss term, It represents the lth negative sample corresponding to the jth positive sample. The number of negative samples in each batch is (N-1)(k+1).
[0044] Preferably, the negative sample mining module in step 4 includes: negative sample mining probability and Bayesian sampling;
[0045] Other images in the training library and samples in the storage library under the same batch are regarded as negative samples;
[0046] D1. Negative sample mining probability: According to the formula:
[0047]
[0048] Among them, s ( q1,z 2) represents the similarity score between the query sample q1 and the positive sample z2, Represents query sample q1 and negative sample The similarity score between represents the probability that the nth negative sample of the query sample q1 is mined, a represents a hyperparameter, and exp represents an exponential function;
[0049] Among them, the greater the similarity between negative samples and positive samples, the higher the probability of being mined;
[0050] D2. The Bayesian sampling: Perform Bernoulli sampling on each negative sample, and finally mine the specific negative sample corresponding to the query sample q1:
[0051] in represents a Bernoulli experiment, accepting z with probability p; represents negative samples, N is the set of negative samples, N mine (q1) represents the set of negative samples mined according to q1. For the query sample q1, the number of negative samples after potential negative sample mining is expressed as |N mine (q1)|;
[0052] Based on this, the complete loss values of the three loss values are obtained:
[0053] in, Indicates a complete loss of this method.
[0054] Preferably, step 5 comprises:
[0055] Get three loss values and Get the total loss value
[0056] The parameters of the online network are updated by gradient by minimizing the total loss value, while the parameters of the target network are updated by momentum update.
[0057] Preferably, step 6 comprises:
[0058] Update the repository with the target network, using a first-in-first-out approach to push the current output batch into the repository while discarding the oldest batch in the queue.
[0059] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the above-mentioned sample mining method based on multi-network self-supervised learning.
[0060] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the above-mentioned sample mining method based on multi-network self-supervised learning are implemented.
[0061] Compared with the prior art, the beneficial effects achieved by the present invention are:
[0062] The sample mining method based on multi-network self-supervised learning proposed in the present invention obtains three views for each image in the data set through data enhancement of different intensities, and different data enhancement intensities bring more diversity to the model; for a positive sample pair, the enhanced sample view and the mined sample view are combined by soft and hard weighting methods to further mine more complex positive samples for the model; for negative samples, high-quality negative samples are mined through gradient probability, and the influence of false negative samples is eliminated, further improving the performance of the model; the present invention can effectively improve the performance of the self-supervised pre-trained network model, and the trained network can improve the image classification accuracy of multiple data sets. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:
[0064] Figure 1 It is a flow chart of a sample mining method based on multi-network self-supervised learning of the present invention;
[0065] Figure 2 It is an algorithm framework of a sample mining method based on multi-network self-supervised learning in the present invention;
[0066] Figure 3 It is a description of the data set involved in the present invention;
[0067] Figure 4 are the values of the parameters involved in the present invention on different data sets;
[0068] Figure 5This is the comparison result of the image classification accuracy of the present invention with different algorithms on three data sets. DETAILED DESCRIPTION
[0069] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0070] See also Figure 1-Figure 5 , the present invention provides a technical solution:
[0071] Embodiment 1:
[0072] A sample mining method based on multi-network self-supervised learning, the method comprising:
[0073] Step 1: Obtain a training set under a certain training batch, perform data enhancement on each image in the training set, and obtain one strongly enhanced image and two weakly enhanced images based on each image;
[0074] The data set includes a training set and a test set; the present invention uses the data of the training set for batch training;
[0075] Preferably, the data enhancement in step 1 includes:
[0076] Perform data augmentation on the images in the training set;
[0077] A1. Weak data augmentation strategies include random cropping and resizing, horizontal flipping, color distortion, and grayscale conversion:
[0078] Define the set of strategies for weak data enhancement as Then for the input image x, according to the formula: Get the transformed image x i ;
[0079] A2. Strong data enhancement strategies include random enhancement and jigsaw operations:
[0080] The strategy set for strong data enhancement is defined as Then for the input image x, according to the formula: Get the transformed image x i .
[0081] Step 2: Input the three images into the corresponding online network and target network respectively to obtain the corresponding image representations;
[0082] Preferably, the online network and the target network in step 2 include:
[0083] The online network and the target network include two parts: an encoder and a projection head, wherein the online network is updated using a gradient, and the target network is updated by an exponential moving average with the help of a back-propagation network.
[0084] Step 3: Consider the strongly enhanced image and the weakly enhanced image of a certain image as positive samples, and the remaining images in the training set as negative samples, and calculate the loss value between the output results of the online network and the projection head to obtain the first loss;
[0085] Preferably, step 3 comprises:
[0086] The strong data enhancement and weak data enhancement of the same image are regarded as a pair of positive samples, and other images are regarded as negative samples;
[0087] Assuming the batch size of training images is N, there are a total of 2N-2 images as negative samples;
[0088] For the obtained strong data enhancement image x3 and weak data enhancement image x1, through the online network and projection head, the projection features q3 and q1 are finally obtained. At this time, the first loss value is calculated according to the loss function
[0089]
[0090] Where exp represents the exponential function, represents negative samples, s represents similarity measure, here cosine similarity is used; t represents temperature hyperparameter, N represents the size of each batch, 2 ( N-1 ) It represents the number of negative samples in each batch.
[0091] Step 4: The two weakly enhanced images of a certain image are regarded as positive samples, and the remaining images in the training set and the samples in the repository are regarded as negative samples. The second loss and the third loss are calculated respectively by combining the positive sample mining module and the negative sample mining module through the online network, the target network and the projection head;
[0092] Preferably, two different weak data enhancements of the same image are regarded as a pair of positive samples, which are respectively passed through the online network and the target network, and other images in the same batch and samples in the repository are regarded as negative samples, and the loss is calculated by feature mining of the positive sample mining module and the negative sample mining module;
[0093] Preferably, the positive sample mining module in step 4 includes: nearest neighbor mining, positive sample hard mining and positive sample soft mining;
[0094] Consider two different weak data enhancements of the same image as a pair of positive samples;
[0095] S1, the nearest neighbor mining: for each sample feature z1 of the online branch, the corresponding sample feature z2 of the target branch is used to mine the expected positive samples in the repository M, and the cosine similarity between samples is used to mine the top k most similar features from the memory library:
[0096]
[0097] where NN ( z 2) It means mining k feature samples that are most similar to z2 from the memory bank M;
[0098] The mining repository M is updated using a first-in-first-out strategy;
[0099] S2, the positive sample hard mining: the enhanced sample view z2 is another feature representation of the same type of sample z1. For the determined positive sample z2, the hard weighting method is adopted, that is, the loss weight w0 is set to 1, and the second loss value is calculated:
[0100]
[0101] Where w0 is the weight coefficient of the loss, z n - represents negative samples, 2 ( N-1 ) is the number of negative samples in the current batch;
[0102] S3, the positive sample soft mining: Considering that there are multiple positive samples, when calculating the contrast loss, the mined samples are not directly regarded as positive samples, but each mined positive sample NN is given ( z 2)i Assign weight w i :
[0103]
[0104] According to NN ( z 2)i The similarity ratio between z1 and the total similarity is used to measure the importance of each mined sample, where
[0105] Get the positive sample soft mining loss value, that is, the third loss value:
[0106]
[0107] where w i represents the weight coefficient of each loss term, It represents the lth negative sample corresponding to the jth positive sample. The number of negative samples in each batch is (N-1)(k+1).
[0108] Preferably, the negative sample mining module in step 4 includes: negative sample mining probability and Bayesian sampling;
[0109] Other images in the training library and samples in the storage library under the same batch are regarded as negative samples;
[0110] D1, the probability of negative sample mining, hoping to mine negative samples that are neither too difficult nor too simple during the training process of contrastive learning: According to the formula:
[0111]
[0112] Among them, s ( q1,z 2) represents the similarity score between the query sample q1 and the positive sample z2, Represents query sample q1 and negative sample The similarity score between Use the differences between them as mining criteria; represents the probability that the nth negative sample of the query sample q1 is mined, a represents a hyperparameter used to control the density of the probability distribution, and exp represents an exponential function;
[0113] Among them, the greater the similarity between negative samples and positive samples, the higher the probability of being mined;
[0114] D2. The Bayesian sampling: Perform Bernoulli sampling on each negative sample, and finally mine the specific negative sample corresponding to the query sample q1:
[0115] in represents a Bernoulli experiment, accepting z with probability p; represents negative samples, N is the set of negative samples, N mine (q1) represents the set of negative samples mined according to q1. For the query sample q1, the number of negative samples after potential negative sample mining is expressed as |N mine (q1)|;
[0116] Based on this, the complete loss values of the three loss values are obtained:
[0117] in, Represents the complete loss of this method. In each iteration, the online network is gradient updated by minimizing the total loss, and then the target network is updated with the help of the online network. During the fine-tuning of the downstream task, only the online network is retained, and an additional linear classification head is added and trained.
[0118] Step 5: Combine the three losses in step 3 and step 4, and perform gradient update on the parameters of the online network and the target network by minimizing the total loss value;
[0119] Preferably, step 5 comprises:
[0120] Get three loss values and Get the total loss value
[0121] The parameters of the online network are updated by gradient by minimizing the total loss value, while the parameters of the target network are updated by momentum update.
[0122] Step 6: Use the target network to update the sample repository. After multiple iterations of training, pre-training is completed.
[0123] Preferably, step 6 comprises:
[0124] Update the repository with the target network, using a first-in-first-out approach to push the current output batch into the repository while discarding the oldest batch in the queue.
[0125] Preferably, Figure 3 The data set involved in the present invention is described, including the name of the data set, the total number of categories, the size of each image, the number of samples in the training set and the test set; use Figure 3 The performance of the proposed scheme and the compared methods are compared on three public datasets in .
[0126] Figure 4 The values of the parameters involved in the present invention on different data sets include training rounds, batch size, learning rate, temperature parameters, etc.
[0127] Figure 5 The image classification accuracy of the present invention is compared with that of different algorithms on three data sets. The top1 and top5 accuracy rates are used to measure the classification accuracy of the image. Figure 5 It can be seen that the correct classification accuracy of the proposed solution MSM (Multi-network Sample Mining) exceeds that of other classic self-supervised methods. In MSM, by introducing more data enhancement views and multiple networks, the diversity of the model can be effectively improved. At the same time, more high-quality negative samples can be provided through the positive and negative sample mining modules.
[0128] Embodiment 2:
[0129] The computer-readable storage medium of this embodiment stores a computer program thereon, and when the program is executed by a processor, the steps in a sample mining method based on multi-network self-supervised learning in embodiment 1 are implemented.
[0130] The computer-readable storage medium of this embodiment may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal; the computer-readable storage medium of this embodiment may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card, a secure digital card, a flash memory card, etc. equipped on the terminal; further, the computer-readable storage medium may also include both an internal storage unit of the terminal and an external storage device.
[0131] The computer-readable storage medium of this embodiment is used to store computer programs and other programs and data required by the terminal. The computer-readable storage medium can also be used to temporarily store data that has been output or is to be output.
[0132] Embodiment 3:
[0133] The computer device of this embodiment includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of a sample mining method based on multi-network self-supervised learning in Embodiment 1 are implemented.
[0134] In this embodiment, the processor may be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, readily available programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.
[0135] Those skilled in the art will appreciate that the disclosed content of the embodiments may be provided as methods, systems, or computer program products. Therefore, the present solution may adopt the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Moreover, the present solution may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program codes.
[0136] The present solution is described with reference to the method according to the embodiment of the present solution and the flowchart and / or block diagram of the computer program product. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions; these computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 one or more processes and / or methods Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0137] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 one or more processes and / or methods Figure 1 A function specified in one or more boxes.
[0138] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 one or more processes and / or methods Figure 1 The steps for the functions specified in one or more boxes.
[0139] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).
[0140] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A sample mining method based on multi-network self-supervised learning, characterized in that: The method comprises: Step 1: Obtain a training set under a certain training batch, perform data enhancement on each image in the training set, and obtain one strongly enhanced image and two weakly enhanced images based on each image; Step 2: Input the three images into the corresponding online network and target network respectively to obtain the corresponding image representations; Step 3: Consider the strongly enhanced image and the weakly enhanced image of a certain image as positive samples, and the remaining images in the training set as negative samples, and calculate the loss value between the output results of the online network and the projection head to obtain the first loss; Step 4: The two weakly enhanced images of a certain image are regarded as positive samples, and the remaining images in the training set and the samples in the repository are regarded as negative samples. The second loss and the third loss are calculated respectively by combining the positive sample mining module and the negative sample mining module through the online network, the target network and the projection head; Step 5: Combine the three losses in step 3 and step 4, and perform gradient update on the parameters of the online network and the target network by minimizing the total loss value; Step 6: Use the target network to update the sample repository. After multiple iterations of training, the pre-training is completed.
2. A sample mining method based on multi-network self-supervised learning as claimed in claim 1, characterized in that: The data enhancement in step 1 includes: Perform data augmentation on the images in the training set; A1. Weak data augmentation strategies include random cropping and resizing, horizontal flipping, color distortion, and grayscale conversion: Define the set of strategies for weak data enhancement as Then for the input image x, according to the formula: Get the transformed image x i ; A2. Strong data enhancement strategies include random enhancement and jigsaw operations: The strategy set for strong data enhancement is defined as Then for the input image x, according to the formula: Get the transformed image x i .
3. The sample mining method based on multi-network self-supervised learning according to claim 1, characterized in that: The online network and the target network in step 2 include: The online network and the target network include two parts: an encoder and a projection head, wherein the online network is updated using a gradient, and the target network is updated by an exponential moving average with the help of a back-propagation network.
4. The sample mining method based on multi-network self-supervised learning according to claim 1, characterized in that: The step 3 comprises: The strong data enhancement and weak data enhancement of the same image are regarded as a pair of positive samples, and other images are regarded as negative samples; Assuming the batch size of training images is N, there are a total of 2N-2 images as negative samples; For the obtained strong data enhancement image x3 and weak data enhancement image x1, through the online network and projection head, the projection features q3 and q1 are finally obtained. At this time, the first loss value is calculated according to the loss function Where exp represents the exponential function, Represents negative samples, s represents the similarity measure, cosine similarity is used here; t represents the temperature hyperparameter, N represents the size of each batch, and 2(N-1) represents the number of negative samples in each batch.
5. The sample mining method based on multi-network self-supervised learning according to claim 1, characterized in that: The positive sample mining module in step 4 includes: nearest neighbor mining, positive sample hard mining and positive sample soft mining; Consider two different weak data enhancements of the same image as a pair of positive samples; S1, the nearest neighbor mining: for each sample feature z1 of the online branch, the corresponding sample feature z2 of the target branch is used to mine the expected positive samples in the repository M, and the cosine similarity between samples is used to mine the top k most similar features from the memory library: Where NN(z2) means mining k feature samples most similar to z2 from the memory bank M; S2, the positive sample hard mining: the enhanced sample view z2 is another feature representation of the same type of sample z1. For the determined positive sample z2, the hard weighting method is adopted, that is, the loss weight w0 is set to 1, and the second loss value is calculated: Where w0 is the weight coefficient of the loss, represents negative samples, 2(N-1) is the number of negative samples in the current batch; S3, the positive sample soft mining: Considering the existence of multiple positive samples, when calculating the contrast loss, the mined samples are not directly regarded as positive samples, but each mined positive sample NN(z2) is given i Assign weight w i : According to NN(z2) i The similarity ratio between z1 and the total similarity is used to measure the importance of each mined sample, where Get the positive sample soft mining loss value, that is, the third loss value: where w i represents the weight coefficient of each loss term, It represents the lth negative sample corresponding to the jth positive sample. The number of negative samples in each batch is (N-1)(k+1).
6. The sample mining method based on multi-network self-supervised learning according to claim 1, characterized in that: The negative sample mining module in step 4 includes: negative sample mining probability and Bayesian sampling; Other images in the training library and samples in the storage library under the same batch are regarded as negative samples; D1. Negative sample mining probability: According to the formula: Among them, s(q1,z2) represents the similarity score between the query sample q1 and the positive sample z2. Represents query sample q1 and negative sample The similarity score between represents the probability that the nth negative sample of the query sample q1 is mined, a represents a hyperparameter, and exp represents an exponential function; Among them, the greater the similarity between negative samples and positive samples, the higher the probability of being mined; D2. The Bayesian sampling: Perform Bernoulli sampling on each negative sample, and finally mine the specific negative sample corresponding to the query sample q1: in represents a Bernoulli experiment, accepting z with probability p; represents negative samples, N is the set of negative samples, N mine (q1) represents the set of negative samples mined according to q1. For the query sample q1, the number of negative samples after potential negative sample mining is expressed as |N mine (q1)|; Based on this, the complete loss values of the three loss values are obtained: in, Indicates a complete loss of this method.
7. The sample mining method based on multi-network self-supervised learning according to claim 1, characterized in that: The step 5 comprises: Get three loss values and Get the total loss value The parameters of the online network are updated by gradient by minimizing the total loss value, while the parameters of the target network are updated by momentum update.
8. The sample mining method based on multi-network self-supervised learning according to claim 1, characterized in that: The step 6 comprises: Update the repository with the target network, using a first-in-first-out approach to push the current output batch into the repository while discarding the oldest batch in the queue.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in a sample mining method based on multi-network self-supervised learning as described in any one of claims 1-8 are implemented.
10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the sample mining method based on multi-network self-supervised learning as described in any one of claims 1-9 are implemented.