A continuous self-supervised learning method for image classification

Abstract

The technical problem to be solved by the present invention is a method for image classification with continuous self-supervised learning. The method uses augmented stability to select samples at the boundaries and centers of the category distribution in each data stream as replay samples to minimize overfitting effects and catastrophic forgetting. Secondly, the method proposes contrast continuity to enable the network to capture as much information shared between data streams as possible to prevent catastrophic forgetting, while removing as much redundant information as possible to improve the network's plasticity and enhance its ability to continuously learn. Compared with random sampling, this method can minimize overfitting effects and catastrophic forgetting. Secondly, the method uses contrast continuity to enable the network to retain as much information shared with past data streams as possible to prevent catastrophic forgetting, while removing redundant information encoded in past data streams to improve the network's plasticity, enhance its ability to continuously learn, and improve image classification performance.

Description

Technical Field

[0001] The present invention relates to a continuous self-supervised learning technology, and in particular to an image classification technology of continuous self-supervised learning. Background Art

[0002] With the development and maturity of deep learning technology in computer vision, image classification methods have essentially reached human-level performance. Standard strongly supervised learning requires collecting large amounts of data and performing time-consuming and labor-intensive manual labeling when training neural networks. However, in real-world scenarios, data is often collected gradually over time, and manual labeling of massive amounts of data is difficult. Therefore, to meet the requirements of real-world image classification methods, we need to develop image classification methods based on continuous self-supervised learning. Its main task is to learn useful feature representations from each data stream in turn, given a given unlabeled data stream. The learned feature representations must not only encode the information of the corresponding data stream, but more importantly, they must not forget the information encoded in the previous data stream. Finally, the learned feature representations must be transferred to the target dataset to complete the image classification task.

[0003] Currently, there are two types of strategies for combating catastrophic forgetting in continuous self-supervised learning: the first type is to select appropriate samples in each data stream as replay samples, and in subsequent data stream training, add the replay samples to the training to prevent catastrophic forgetting; the second type is to add some regularization techniques, such as knowledge distillation, to the network optimization objectives, so that the network can maintain some consistency with the previous state in subsequent data stream training. In the first type of method, because the replay samples are just samples of the past data stream, it is easy for the network to overfit to the replay samples, resulting in catastrophic forgetting; in the second type of method, because the regularization term forces the network to maintain consistency with the previous state when learning the current data stream, it is easy to limit the network's plasticity, making it unable to effectively encode the information of the current data stream. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method for providing better image classification effect by improving the continuous learning ability of continuous self-supervised learning and reducing catastrophic forgetting.

[0005] The technical solution adopted by the present invention to solve the above technical problems is a continuous self-supervised learning image classification method. The method uses augmented stability to select samples at the boundary and center of the category distribution in each data stream as replay samples to minimize overfitting and catastrophic forgetting. Secondly, contrast continuity is proposed to enable the network to capture as much shared information between data streams as possible to prevent catastrophic forgetting, while removing as much redundant information as possible to improve the network plasticity and enhance its ability to continue learning. The method specifically includes the following steps:

[0006] Step 1: Initialize the feature extraction network and use the loss function Constraining the self-supervised contrastive learning of the first image data stream to complete the initial training of the feature extraction network;

[0007] Among them, cos is cosine similarity, h is the prediction head, f θ is the current feature extraction network, and Two different augmentation strategies are used, where x is the input image sample; the feature extraction network outputs paired features z1 and z2; the first image data stream is the input image sample; the trained image data stream is used as the trained image data stream;

[0008] Step 2: Use the discriminator to infer the augmented stability score of each sample in the data stream, and sort the augmented stability of the trained image data stream from large to small according to the augmented stability score;

[0009] Step 3: Select the samples with the highest ranking as the replay samples;

[0010] Step 4: The replay memory is updated by discarding the replay samples in the middle of the augmented stability ranking of each task in the replay memory and then storing the latest replay samples;

[0011] Step 5: Use contrastive continuity loss Combined with the loss function Constrain the self-supervised contrastive learning of the current image data stream to complete the training of the current feature extraction network;

[0012]

[0013] , where t is the current image data stream number, τ is the image data stream number τ=1,...,t-1, D t represents t image data streams, is the past state of the feature extraction network, is the replay sample, β is the hyperparameter;

[0014] Step 6: If the trained image data stream is the last image data stream, perform classification layer training to complete the image classification model training; otherwise, execute step 2.

[0015] Specifically, the augmented stability score of each sample is: p(y='0'|x)=p D (y='0'|(z1,z2)), where y is the judgment category, '0' is the augmented stable type, '1' is the augmented unstable type, p(y='0'|x) represents the probability of augmented stability of the input sample x, that is, the augmented stability score, p D (y='0'|(z1,z2)) represents the probability that the input pair of features z1 and z2 is judged as '0' by the discriminator D.

[0016] Specifically, the specific steps of classification layer training are:

[0017] Step 7: Add a linear classification layer to the trained feature extraction network;

[0018] Step 8: Fine-tune the linear classification layer using the target image dataset to complete the image classification training task.

[0019] The main innovation of the invention is to use augmented stability to estimate the relative position of samples in the category distribution, and use samples at the center and edge of the distribution as replay samples. Compared with random sampling, this method can minimize the overfitting effect and catastrophic forgetting; secondly, it uses contrastive continuity to enable the network to retain as much information as possible shared with past data streams to prevent catastrophic forgetting, while eliminating redundant information encoded in past data streams to improve network plasticity and enhance its ability to continue learning.

[0020] The beneficial effects of the present invention are that augmented stability replay can select samples at the center and boundary of the category distribution in each data stream, thereby minimizing the overfitting effect and catastrophic forgetting; contrast continuity can improve the plasticity of the network on the basis of preventing catastrophic forgetting, enhance its ability of continuous learning, and improve image classification performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart of an embodiment;

[0022] Figure 2 Updated schematic for replay memory. DETAILED DESCRIPTION

[0023] Given T data streams D 1:T =(D1,…,D T ), each data stream Contains n tsamples, each sample has no corresponding manual label, and the data stream sequence number is t∈{1,…,T}. Continuous self-supervised learning is dedicated to learning a feature extraction network f θ It captures information that is beneficial to downstream tasks in each data stream in turn, while not forgetting the information encoded in the previous data stream, and finally migrates the learned feature representation to the target dataset to complete the image classification task.

[0024] The present invention is based on the classic self-supervised contrastive learning algorithm, which encourages the augmentation invariance of samples through a prediction head h(·). Its loss function is as follows:

[0025]

[0026] Where cos(A,B) means calculating the cosine similarity of two vectors A and B:

[0027]

[0028] is a standard self-supervised data augmentation strategy. We use different superscripts to distinguish different augmentation strategies, namely and It can be called the first augmentation strategy or the second augmentation strategy.

[0029] One approach in continuous self-supervised learning involves selecting appropriate samples as replay samples after training each data stream and incorporating them into the training of subsequent data streams to prevent catastrophic forgetting. However, since replay samples are simply samples of the past data stream distribution, they can easily cause the network to overfit to them, leading to catastrophic forgetting. To address this overfitting effect of replay samples, we propose a replay sample selection strategy based on augmented stability. Generally speaking, we should select the most representative and discriminative samples for replay, namely those located at the boundaries and center of each class distribution. These samples retain most of the information from the previous task, significantly mitigating overfitting and combating catastrophic forgetting. However, in the unsupervised setting, we cannot know the class of the sample and therefore cannot determine its relative position in the corresponding class distribution. To overcome this difficulty, the applicants discovered that the augmented stability of each sample is positively correlated with its relative position in the corresponding class distribution. Therefore, we use augmented stability to estimate the relative position of the sample in the corresponding class distribution and select samples located at the center and boundaries of the distribution for replay.

[0030] We designed a discriminator D(·) to estimate the augmented stability of the sample. It is essentially a binary classifier that takes the paired features output by the self-supervised model as input and outputs the judgment result of whether the input paired features come from the same image. During the training process, the loss of the discriminator The build is as follows:

[0031]

[0032] Where CE represents the cross entropy loss between A and B, and the intermediate Overall, the loss The constrained discriminator D(·) judges the paired features z1 and z2 from the same augmented view as class ‘0’, and the paired features z1 and z2 from different augmented views as class ‘0’. It is judged as class '1', thereby obtaining the ability to estimate the augmented stability of the sample.

[0033] When we need to select replay samples from a trained data stream, we first use the discriminator to infer the augmented stability score for each sample in the data stream:

[0034] p(y='0'|x)=p D (y='0'|(z1,z2)) (4)

[0035] Among them, x is the input sample, y is the judgment category, '0' is the augmented stable type, '1' is the augmented unstable type, p(y='0'|x) represents the probability of augmented stability of the input sample x, that is, the augmented stability score, p D (y='0'|(z1,z2)) represents the probability that the input pair of features z1 and z2 is discriminated as '0' by the discriminator D(·).

[0036] Then, all samples are sorted using the augmented stability score, and the samples at the front (high augmented stability, located in the center of the class distribution, representative) and the samples at the back (low augmented stability, located at the boundary of the class distribution, discriminative) are selected as replay samples according to the sorted list.

[0037] In addition, we also develop a matching update strategy to dynamically update the replay memory, the update diagram is shown in Figure 2 Specifically, when storing the latest data stream, we first recalculate the same amount of storage space for all known tasks. We then update the replay memory by discarding the replay samples with the middle augmented stability ranking for each task in the replay memory and then storing the latest replay samples. These discarded samples are considered the least representative or discriminative samples.

[0038] Continuous self-supervised learning requires retaining information encoded in past data streams to prevent catastrophic forgetting, while also encoding information in the current data stream to improve the ability to continuously learn. In practice, all the information encoded in past data streams is not only redundant for preventing catastrophic forgetting, but also hinders the model's learning of the current data stream. To remove redundant information encoded in past data streams and balance the prevention of catastrophic forgetting with the development of continuous learning capabilities, we further propose contrastive continuity based on the Information Bottleneck (IB) principle. Its core purpose is to retain as much shared information between known data streams as possible to prevent catastrophic forgetting, while removing redundant information to unleash the ability to continuously learn.

[0039] The IB principle states that an ideal representation Z should provide as much important information as possible about the label Y, while compressing the original information from the input X by eliminating redundant parts:

[0040] IB=I(Z;X)-βI(Z;Y) (5)

[0041] Where I(A; B) represents the mutual information between A and B, and β is a hyperparameter used to balance the important information retained and the redundant parts eliminated.

[0042] Inspired by the IB principle, the applicant will contrast the continuity expectation model to encode as much information shared with past data streams as possible to prevent catastrophic forgetting, and eliminate the redundant information encoded in past data streams to release the ability of continuous learning. Specifically, given the current data stream D t and the corresponding replay memory B t-1 , where we use represents the replay sample of the τth (τ=1,...,t-1)th data stream in the replay memory. Contrast continuity is committed to constraining the current model f θ and past state In the corresponding replay sample To capture the information shared with past data streams to prevent catastrophic forgetting, and to maintain consistency in Constrain the current model f θ and past state In the current data stream sample D t Maintaining differences in order to remove redundant information encoded in past data streams and develop the ability to continuously learn:

[0043]

[0044] Obviously, Start working when t>1. At the same time, the current There is also a problem of imbalance between the number of current data stream samples and replay samples, i.e. Therefore, in practice, from t image data streams D t Sampling a subset To replace the current image data stream to train the current feature extraction network, subset The number of samples and replay samples in The number of samples is the same, To solve the problem of quantity imbalance and significantly reduce The computational complexity of .

[0045] The embodiment is implemented on the Pytorch deep learning framework, such as Figure 1 The main steps shown are: estimating the augmented stability ranking of the data stream, selecting replay samples based on the ranking, updating the replay memory, and constructing contrastive continuity loss to continue training.

[0046] Step 1: Feature extraction network training steps

[0047] Step 1: Initialize the image classification model and perform self-supervised training on the first image data stream using formula (1);

[0048] Step 2: Estimate the augmented stability ranking of the trained image data stream using formula (4);

[0049] Step 3: Select replay samples based on the ranking;

[0050] Step 4: Figure 2 The update of the replay memory is shown as follows, that is, recalculating the same amount of storage space for all known tasks, then discarding the redundant samples in the middle of the replay samples sorted by augmented stability for each task, and finally storing the latest replay samples. For example, Figure 2 Middle, B t Represents the replay memory after learning the data stream of t tasks, that is, it stores the replay samples of t data streams. Figure 2are represented by different textures. The replay memory has 24 blocks of storage space. The replay memory B1 of the first stage uses 24 blocks of data to store the replay samples of task T1. If there is new data of task T2 to be stored, B1 needs to be updated. At this time, the same amount of storage space is recalculated for the stored previous task T1 and the current task T2. To store the 12 blocks of data of the current task T2, the 12 blocks of data in the previous task T1 need to be discarded. Preferably, the middle samples in the replay samples sorted by augmented stability of each task are discarded, that is, the 12 blocks of data in the middle position in B1 are discarded first, and then the 12 blocks of data of the current task T2 are stored to obtain the replay memory B2 of the second stage. The first 12 blocks of data of B2 are the replay samples of task T1, and the last 12 blocks of data are the replay samples of task T2. When new data for task T3 needs to be stored, the storage space for tasks T1, T2, and T3 needs to be recalculated. To store the 8 blocks of data for the current task T3, the 8 blocks of data from tasks T1 and T2 need to be discarded. Similarly, the data in the middle position after sorting T1 and T2 is discarded, resulting in the third-stage replay memory B3. The first 8 blocks of data in B3 are the replay samples of task T1, the middle 8 blocks of data are the replay samples of task T2, and the last 8 blocks of data are the replay samples of task T3. When the data for the new task T4 needs to be stored, the 2-bit data in the middle position after sorting T1, T2, and T3 are discarded and the 6 blocks of data from task T4 are stored. And so on.

[0051] Step 5: Construct the contrast continuity loss as in formula (6) and combine it with the original self-supervised loss to perform self-supervised training on the next image data stream;

[0052] Step 6: If the trained image data stream is the last image data stream, go to step 7; otherwise, go to step 2.

[0053] Step 2: Classification layer training steps

[0054] Step 7: Add a linear classification layer to the trained feature extraction network;

[0055] Step 8: Fine-tune on the target image dataset to complete the classification task.

Claims

1. A continuous self-supervised learning image classification method, characterized in that The following steps are involved: Step 1: Initialize the feature extraction network and use the loss function Constraining the self-supervised contrastive learning of the first image data stream to complete the initial training of the feature extraction network; Among them, cos is cosine similarity, h is the prediction head, f θ is the current feature extraction network, and Two different augmentation strategies are used, where x is the input image sample; the feature extraction network outputs paired features z1 and z2; the first image data stream is the input image sample; the trained image data stream is used as the trained image data stream; Step 2: Use the discriminator to infer the augmented stability score of each sample in the data stream, and sort the augmented stability of the trained image data stream from large to small according to the augmented stability score; Step 3: Select the samples with the highest ranking as the replay samples; Step 4: The replay memory is updated by discarding the replay samples in the middle of the augmented stability ranking of each task in the replay memory and then storing the latest replay samples; Step 5: Use contrastive continuity loss Combined with the loss function Constrain the self-supervised contrastive learning of the current image data stream to complete the training of the current feature extraction network; Where t is the current image data stream sequence number, τ is the image data stream sequence number τ=1,...,t-1, D t represents t image data streams, is the past state of the feature extraction network, is the replay sample, β is the hyperparameter; Step 6: If the trained image data stream is the last image data stream, perform classification layer training to complete the image classification model training; otherwise, execute step 2.

2. The method according to claim 1, wherein: The augmented stability score of each sample is: p(y='0'|x)=p D (y='0'|(z1,z2)), where y is the judgment category, '0' is the augmented stable type, '1' is the augmented unstable type, p(y='0'|x) represents the probability of augmented stability of the input sample x, that is, the augmented stability score, p D (y='0'|(z1,z2)) represents the probability that the input pair of features z1 and z2 is judged as '0' by the discriminator D.

3. The method according to claim 1, wherein: The specific steps of classification layer training are: Step 7: Add a linear classification layer to the trained feature extraction network; Step 8: Fine-tune the linear classification layer using the target image dataset to complete the image classification training task.

4. The method according to claim 1, wherein: In step 5, from t image data stream D t Sampling a subset To replace the current image data stream to train the current feature extraction network, subset The number of samples and replay samples in The number of samples is the same,

Images