An interpretable self-supervised pre-training method based on feature decoupling

Abstract

The present invention provides an interpretable self-supervised pre-training method based on feature decoupling. By introducing an importance matrix into the loss function of self-supervised contrastive learning, feature decoupling is achieved, and feature importance ranking is obtained according to the size of the diagonal elements of the importance matrix, thereby improving the interpretability of the self-supervised pre-training algorithm. The method comprises the following steps: initializing a neural network; initializing an importance matrix; training the neural network m times, each training including: sampling a data set, performing data augmentation on the samples, encoding features using the neural network, constructing positive and negative sample pairs, calculating a loss function, and using a gradient backpropagation algorithm to calculate the gradient of the loss function with respect to the neural network and the importance matrix, and updating the neural network and the importance matrix; after m times of iterative training, the features encoded and output by the neural network are decoupled from each other, and the larger the value of the diagonal element of the feature importance matrix, the more important the corresponding feature dimension, thereby obtaining a feature importance ranking.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning model pre-training technology in artificial intelligence, and relates to self-supervised pre-training technology, specifically to an interpretable self-supervised pre-training method based on feature decoupling. It is a self-supervised pre-training method that is universal for different neural network models and different modal data. Background Art

[0002] With the introduction of deep neural networks such as AlexNet and ResNet, machine learning has entered the era of deep learning. However, traditional supervised learning algorithms require large amounts of labeled data to train deep neural networks, making accurately labeling this massive amount of data a challenging task. To overcome this challenge, researchers have proposed self-supervised learning algorithms that do not require data labeling. Through continuous improvements to self-supervised training algorithms, these algorithms are rapidly approaching and even surpassing the performance of supervised learning in fields such as image recognition, natural language processing, and speech recognition. Contrastive learning is a representative method in the field of self-supervised learning. Within this framework, researchers transform the same image into different augmented perspectives through hand-crafted data transformations such as cropping, flipping, and color dithering. Subsequently, by training a contrastive loss function, such as InfoNCE, the different augmented perspectives of the same image (positive samples) are brought closer together in feature space while simultaneously pushing the augmented perspectives of different images (negative samples) further apart, resulting in a deep neural network with excellent performance. At present, as a general pre-training method, representative algorithms of contrastive learning such as SimCLR and CLIP have demonstrated unique advantages and surpassed the performance of supervised learning in multiple downstream task scenarios such as small sample classification, transfer learning, and cross-modal retrieval.

[0003] While self-supervised learning has demonstrated exceptional performance on a variety of downstream tasks, most current self-supervised algorithms are heuristic and research into their working mechanisms is still exploratory. Consequently, the interpretability of self-supervised features is poor, hindering their performance in tasks that rely on feature interpretability, such as low-dimensional image feature retrieval, causal inference, and out-of-distribution generalization. Therefore, improving the interpretability of self-supervised pre-trained features is a pressing issue. To address this issue, researchers have recently explored the underlying reasons for the difficulty in analyzing self-supervised features through theoretical characterizations of self-supervised training. Tengyu Ma's team at Stanford University pointed out that the optimal representations obtained by current self-supervised learning algorithms contain a variable matrix, which means that different feature dimensions are coupled together, preventing feature decoupling. Feature decoupling is a crucial metric for measuring the transferability and interpretability of representation learning. A lack of feature decoupling can lead to numerous problems. For example, the coupled features in self-supervised learning representations make it impossible to rank the importance of different feature dimensions. Current research at Duke University indicates that the presence of a large number of feature dimensions in pre-trained models during downstream migration can impair their ability to perform downstream migration. Therefore, it is necessary to screen the importance of feature dimensions. In supervised learning, researchers use algorithms such as gradient information within the same category to rank feature importance, thereby selecting dimensions that are beneficial for downstream migration tasks. However, because current algorithms have difficulty ranking coupled features, these methods cannot be transferred to self-supervised algorithms, which greatly limits the generalization ability of self-supervised pre-trained features.

[0004] Therefore, to address the current problem of poor interpretability of self-supervised pre-training features, a primary goal is to design a self-supervised pre-training method that achieves feature decoupling. However, due to the lack of supervised information and the unique nature of self-supervised pre-training, the feature decoupling training algorithms used in traditional supervised learning are no longer applicable. Therefore, designing an interpretable self-supervised pre-training method that achieves feature decoupling is of great significance. Summary of the Invention

[0005] In response to the problems existing in the above-mentioned prior art, the present invention proposes an interpretable self-supervised pre-training method based on feature decoupling. By improving the pre-training loss function of self-supervised learning, the mutual decoupling of pre-training features is achieved, and the importance ranking of different dimensions of self-supervised pre-training features is obtained, thereby improving the interpretability and transferability of self-supervised pre-training features, thereby enabling self-supervised pre-training features to perform better in real-world application scenarios such as low-dimensional feature retrieval of images, causal inference, and out-of-distribution generalization.

[0006] The technical solutions of the present invention are as follows:

[0007] An interpretable self-supervised pre-training method based on feature decoupling is proposed. This method introduces an importance matrix into the loss function for self-supervised contrastive learning, absorbs free variables in the optimal self-supervised feature representation, achieves feature decoupling, and derives feature importance ranking based on the size of the diagonal elements of the importance matrix, thereby improving the interpretability of the self-supervised pre-training algorithm.

[0008] Define the following parameters:

[0009] The neural network is denoted as f, the output dimension of the neural network encoder is denoted as d, the importance matrix is denoted as S, the number of neural network training iterations is m, the number of samples sampled in each training is n, and a single sample is denoted as x i , the n samples sampled are recorded as The 2n samples after two data enhancements are recorded as After 2n samples are encoded by neural network features, the set of 2n features output is expressed as In the process of calculating the loss function, the 2n features in the feature set are paired with each other to obtain a single positive sample pair denoted as (f(x i1 ),f(x i2 )), we get n positive sample pairs denoted as (f(x),f(x + ), randomly selected from the data set with x i Unrelated samples are recorded as Sample n and Unrelated samples are recorded as The features output by neural network feature encoding are recorded as After neural network feature encoding, the set of n features output is recorded as Will n features and The 2n features are paired separately to obtain a single negative sample pair denoted as The 2n negative sample pairs are denoted as The mathematical expectation of the positive sample pair is recorded as E x , the mathematical expectation of the negative sample pair is recorded as The identity matrix is denoted as I, and the optimized neural network is denoted as f ＊ , the optimized feature importance matrix is recorded as S ＊ ;

[0010] The method specifically comprises the following steps:

[0011] S1. Initialize the neural network f and set the feature dimension of the neural network encoder output to d;

[0012] S2. Initialize the importance matrix S. The dimension of S matches the dimension d of the output features of the neural network f. Its off-diagonal elements are 0 and the diagonal elements are random non-negative numbers.

[0013] S3. Train the neural network for m iterations. If the number of iterations exceeds each iteration, perform the following operations:

[0014] S31. Take n samples from the data set For each sample x i , perform two data enhancements, and get a total of 2n samples At the same time, randomly select the same i Unrelated samples Take n unrelated samples and record them as

[0015] S32. Samples after data enhancement After feature encoding by neural network, a set of 2n features is obtained At the same time After neural network feature encoding, a set of n features is obtained

[0016] S33. Construct positive sample pairs and transform 2n features Paired two by two, the positive sample pair of a single sample is recorded as (f(x i1 ),f(x i2 )), a total of n positive sample pairs can be obtained, denoted as (f(x),f(x + )), and construct negative sample pairs at the same time, n features and The 2n features are paired separately to obtain the negative sample pair of a single sample, which is recorded as A total of 2n negative sample pairs can be obtained, recorded as (f(x),f(x - ));

[0017] S34. Calculate the loss function:

[0018]

[0019] Among the three major terms on the right side of the equation, the first one measures the similarity of the positive sample pairs, the second one measures the dissimilarity of the negative sample pairs, and the third one is a regularization term to ensure that the features are orthogonal, where E x It is the mathematical expectation of the positive sample pair, is the mathematical expectation of the negative sample pair, I is the d-dimensional identity matrix, and the loss function value L is calculated by equation tri (f,S);

[0020] S35. Use the gradient backpropagation algorithm to calculate the gradient of the loss function with respect to the neural network parameters and the importance matrix S, that is, and During the optimization process, the gradient backpropagation algorithm is combined with the optimization method to update the neural network f and the importance matrix S;

[0021] S4. After m iterations of training, the optimized neural network f is obtained ＊ And the feature importance matrix S ＊ ; Among them, through f ＊ The encoded features are decoupled from each other, and S ＊ The diagonal elements represent the importance of the corresponding feature dimension. The larger the value of the diagonal element, the more important the corresponding dimension. According to S ＊ The size of the diagonal elements is ranked by feature importance.

[0022] Furthermore, the step S1 initializes the neural network f, and the operation includes: using the Python programming language to construct an 18-layer deep convolutional neural network, and using a deep learning toolkit to randomly initialize the parameter values of the neural network f.

[0023] Furthermore, the step S2 initializes the non-zero and non-negative diagonal elements in the importance matrix S to random numbers uniformly distributed in the interval (0, 1).

[0024] Furthermore, the number of training times in step S3 is determined based on the complexity of the problem and available computing resources.

[0025] Furthermore, the data set in step S31 uses the CIFAR-10 data set, and the data enhancement operations include random cropping, scaling, flipping, color jittering, grayscale change and Gaussian blur processing.

[0026] Furthermore, in step S32, the data-enhanced samples are subjected to feature encoding by a neural network, and forward propagation of the network is used to obtain their feature representations.

[0027] Furthermore, in the process of calculating the loss function in step S34, the gradient clipping method is first used to avoid updating the non-diagonal elements of the importance matrix S, so that the non-diagonal elements remain 0. At the same time, an activation function Sigmoid is applied to the diagonal elements of the importance matrix S to keep the diagonal elements non-negative, and then the calculation of the loss function is entered.

[0028] Furthermore, the step S35 optimization method uses the layered adaptive rate scaling algorithm LARS algorithm, whose setting parameters include the initial value of the learning rate, the learning rate update strategy and the weight decay parameter. The initial learning rate is set to 4.0, the learning rate is updated using the cosine decay strategy and the weight decay parameter is set to 0.000001.

[0029] Compared with the prior art, the technical effects of the present invention are as follows:

[0030] The present invention provides an interpretable self-supervised pre-training method based on feature decoupling. By introducing the feature importance matrix into the loss function of self-supervised contrastive learning, it can decouple the self-supervised features and adaptively obtain the importance ranking of feature dimensions, thereby greatly improving the interpretability of the self-supervised features, and has potential applications in multiple fields. In medical diagnosis and prediction, interpretable features can help doctors understand the basis of the model's disease prediction, which is of great significance for formulating treatment plans and making decisions, and can also provide patients with better communication and understanding. In fields such as image classification and target detection, interpretable features can reveal the model's focus on different objects and features, helping people understand the working principles of the model in visual tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A flowchart of the interpretable self-supervised pre-training method based on feature decoupling provided by the present invention; DETAILED DESCRIPTION

[0032] The present invention will be further clearly and completely described below through specific embodiments in conjunction with the accompanying drawings.

[0033] Figure 1 This is a flowchart of an interpretable self-supervised pre-training method based on feature decoupling of the present invention, which specifically includes the following steps:

[0034] S1. Initialize the neural network f according to the requirements of the problem and the architecture of the model, including defining the type, depth, scale, etc. of the neural network, and randomly initialize the neural network parameter values.

[0035] S2. Create an importance matrix S. The dimension of this matrix should match the output feature dimension d of the neural network f. In order to achieve decoupling and interpretability of the features, the non-diagonal elements of the matrix are initialized to zero. At the same time, the diagonal elements are initialized to random non-negative numbers uniformly distributed in the interval (0, 1).

[0036] S3. Set the number of iterations m, which usually needs to be determined based on the complexity of the problem and the available computing resources.

[0037] S31. In each iteration, n samples need to be randomly selected from the dataset For each sample x i , perform two different data enhancement operations, and get a total of 2n samples Data augmentation operations can be selected based on the nature of the task. For example, in image processing tasks, operations such as cropping, scaling, and flipping can be included. At the same time, random data with the same value as x is selected from the dataset. i Unrelated samples Take n unrelated samples and record them as

[0038] S32. Next, the samples generated by data augmentation are input into the neural network f to obtain their feature representations. After feature encoding through neural network f, a set of 2n features is obtained At the same time After neural network f feature encoding, a set of n features is obtained These feature representations will serve as input in subsequent steps to calculate the loss function and update the neural network parameters.

[0039] S33. Construct positive and negative sample pairs, and transform 2n features Paired two by two, the positive sample pair of a single sample is recorded as (f(x i1 ),f(x i2 )), these two samples come from two different data-enhanced versions of the same original sample, and a total of n positive sample pairs can be obtained, denoted as (f(x),f(x + )), and construct negative sample pairs at the same time, n features and The 2n features are paired separately to obtain the negative sample pair of a single sample, which is recorded as A total of 2n negative sample pairs can be obtained, recorded as (f(x),f(x - )).

[0040] S34. Calculate the loss function:

[0041]

[0042] This loss function has three terms: the first term measures the similarity of positive sample pairs, the second term measures the dissimilarity of negative sample pairs, and the third term is a regularization term to ensure that the features are orthogonal, where E x It is the mathematical expectation of the positive sample pair, is the mathematical expectation of the negative sample pair, I is the d-dimensional identity matrix, and the loss function value L is calculated by equation tri (f,S).

[0043] S35. Finally, the gradient back propagation algorithm is used to calculate the gradient of the loss function with respect to the neural network f parameters and the importance matrix S, that is, and During the optimization process, an optimization method (such as the layered adaptive rate scaling algorithm (LARS), the stochastic gradient descent algorithm, etc.) is selected and the corresponding parameters are determined, including the initial value of the learning rate and the learning rate update strategy. Then, the gradient backpropagation algorithm is combined with the optimization method to update the neural network f and the importance matrix S.

[0044] S4. After multiple iterations of training, the neural network f is obtained after being adjusted during the optimization process. ＊ And the feature importance matrix S ＊ , through f ＊ The encoded features are decoupled from each other, and the feature importance matrix S ＊ The diagonal elements carry the importance information about each feature dimension. The larger the value of the diagonal element, the higher the importance of the corresponding dimension. According to the feature importance matrix S ＊ The size of the diagonal elements is ranked by feature importance.

[0045] Example 1:

[0046] This embodiment assumes that an 18-layer convolutional neural network model (ResNet-18) is used to process the CIFAR-10 dataset (a total of ten classes, 5000 images in each class, and the size of each image is 32×32) to improve the performance of self-supervised pre-training in downstream low-dimensional feature retrieval tasks. During the self-supervised pre-training process, 1024 samples are randomly selected from the dataset for each training step. A total of 50,000 training times are performed. In the process of gradient backpropagation, specifically, the layered adaptive rate scaling algorithm (LARS) is selected to update the learning rate using the cosine decay strategy. The initial learning rate is set to 4.0, and the weight decay parameter is set to 0.000001. The specific steps are as follows:

[0047] S1. First, use the Python programming language to build code and create an 18-layer convolutional neural network model (ResNet-18). The model is loaded onto a GeForce RTX 3090 graphics processing unit (GPU). The entire neural network f is randomly initialized using the deep learning toolkit. The output feature dimension of the neural network is 512×512.

[0048] S2. Use the Deep Learning Toolkit to create a tensor representing the importance matrix S. The dimensions of matrix S are equal to the output feature dimension of the neural network, 512×512. To ensure feature decoupling and interpretability, initialize the matrix so that all off-diagonal elements are zero and all diagonal elements are non-negative. Therefore, first set the off-diagonal elements of the matrix to 0, then use the Deep Learning Toolkit to generate random numbers from a uniform (0, 1) distribution to initialize the diagonal elements to random non-negative numbers.

[0049] S3. Set the number of iterations to 50,000 and perform the following operations in each iteration:

[0050] S31. In each iteration, randomly select 1024 samples from the CIFAR-10 training set and for each sample x i Perform two different data enhancements to obtain a total of 2048 samples Each data augmentation operation includes random cropping and scaling with a cropping ratio of (0.08, 1), color jittering with a brightness, contrast, saturation, and hue variation intensity of (0.8, 0.8, 0.8, 0.2), grayscale variation with a probability of 0.8, and Gaussian blur. At the same time, samples that are not related to the selected samples are randomly selected from the CIFAR-10 training set. Get 1024 samples

[0051] S32. Next, the samples generated by data enhancement Input into the convolutional neural network ResNet-18 and pass through the network's forward propagation to obtain their feature representation set Similarly, samples that are not related to it It is also input into the convolutional neural network ResNet-18, and their feature representation set is obtained through the forward propagation of the network These feature representations will serve as input to subsequent steps to calculate the loss function and update the network parameters.

[0052] S33. Construct positive and negative sample pairs, for each sample x i , construct positive sample pairs (f(x i1 ),f(x i2 )), a total of 1024 positive sample pairs are constructed, denoted as (f(x),f(x + )); At the same time, for each sample Constructing negative sample pairs A total of 2048 negative sample pairs are constructed, denoted as (f(x),f(x - )).

[0053] S34. Calculate the loss function:

[0054]

[0055] During the calculation process, to ensure that the off-diagonal elements of the importance matrix remain zero, a gradient clipping method is used to avoid updating the off-diagonal elements. At the same time, to ensure that the diagonal elements of the importance matrix are non-negative, a sigmoid activation function is applied to the diagonal elements of the importance matrix before entering the loss function calculation.

[0056] S35. Finally, the gradient back propagation algorithm is used to calculate the gradient of the loss function with respect to the neural network parameters and the importance matrix S, that is, and During the optimization process, the layered adaptive rate scaling algorithm (LARS) was selected to update the learning rate using the cosine decay strategy. The initial learning rate was set to 4.0, and the weight decay parameter was set to 0.000001 to update the neural network f and the importance matrix S.

[0057] S4. After 50,000 iterations of training, the neural network f is obtained after being adjusted during the optimization process. ＊ And the feature importance matrix S ＊ , f ＊ The features of the encoded output are decoupled from each other, and the feature dimensions of the samples are divided according to S ＊ The corresponding diagonal elements are sorted by their size. The larger the corresponding diagonal element is, the more important the feature dimension is considered.

[0058] After obtaining the feature importance ranking, low-dimensional feature retrieval is performed on the CIFAR-10 dataset. Specifically, according to the importance matrix, the 40 most important features are selected from the 512-dimensional features of the neural network. Then for each picture, 50 nearest neighbors are found in the feature space, and it is determined whether they belong to the same category as the picture to calculate the accuracy. At the same time, a ResNet-18 is trained on CIFAR-10 using the current representative method of contrastive learning, and 40 dimensions are randomly selected from its 512-dimensional features to perform low-dimensional feature retrieval of pictures on CIFAR-10. Finally, the performance of the original self-supervised method and the decoupling-based interpretable self-supervised pre-training method proposed in the present invention on this task is compared, and it is found that the retrieval accuracy is significantly improved.

[0059] Finally, it should be noted that the purpose of disclosing the embodiments is to facilitate a further understanding of the present invention. However, those skilled in the art will appreciate that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the contents disclosed in the embodiments, and the scope of protection claimed by the present invention shall be determined by the scope defined in the claims.

Claims

1. An interpretable self-supervised pre-training method based on feature decoupling, characterized in that: The importance matrix is introduced into the self-supervised contrastive learning loss to absorb the free variables in the optimal self-supervised feature representation, achieve feature decoupling, and rank the feature importance based on the size of the diagonal elements of the importance matrix, improving the interpretability of the self-supervised pre-training algorithm. Define the following parameters: The neural network is denoted as f, the output dimension of the neural network encoder is denoted as d, the importance matrix is denoted as S, the number of neural network training iterations is m, the number of samples sampled in each training is n, and a single sample is denoted as x i , the n samples sampled are recorded as The 2n samples after two data enhancements are recorded as After 2n samples are encoded by neural network features, the set of 2n features output is expressed as In the process of calculating the loss function, the 2n features in the feature set are paired with each other to obtain a single positive sample pair denoted as (f(x i1 ),f(x i2 )), we get n positive sample pairs denoted as (f(x),f(x + ), randomly selected from the data set with x i Unrelated samples are recorded as Sample n and Unrelated samples are recorded as The features output by neural network feature encoding are recorded as After neural network feature encoding, the set of n features output is recorded as Will n features and The 2n features are paired separately to obtain a single negative sample pair denoted as 2n negative sample pairs are recorded as (f(x),f(x - )), the mathematical expectation of the positive sample is recorded as E x , the mathematical expectation of the negative sample pair is recorded as The identity matrix is denoted as I, the final optimized neural network is denoted as f*, and the optimized feature importance matrix is denoted as S*; The method specifically comprises the following steps: S1. Initialize the neural network f and set the feature dimension of the neural network encoder output to d; S2. Initialize the importance matrix S. The dimension of S matches the dimension d of the output features of the neural network f. Its off-diagonal elements are 0 and the diagonal elements are random non-negative numbers. S3. Train the neural network for m iterations, performing the following operations in each iteration: S31. Take n samples from the data set For each sample x i , perform two data enhancements, and get a total of 2n samples At the same time, randomly select the same i Unrelated samples Take n unrelated samples and record them as S32. Samples after data enhancement After feature encoding by neural network, a set of 2n features is obtained At the same time After neural network feature encoding, a set of n features is obtained S33. Construct positive sample pairs and transform 2n features Paired two by two, the positive sample pair of a single sample is recorded as (f(x i1 ),f(x i2 )), a total of n positive sample pairs can be obtained, denoted as (f(x),f(x + )), and construct negative sample pairs at the same time, n features and The 2n features are paired separately to obtain the negative sample pair of a single sample, which is recorded as A total of 2n negative sample pairs can be obtained, recorded as (f(x),f(x - )); S34. Calculate the loss function: Among the three major terms on the right side of the equation, the first one measures the similarity of the positive sample pairs, the second one measures the dissimilarity of the negative sample pairs, and the third one is a regularization term to ensure that the features are orthogonal, where E x It is the mathematical expectation of the positive sample pair, is the mathematical expectation of the negative sample pair, I is the d-dimensional identity matrix, and the loss function value L is calculated by equation tri (f,S); S35. Use the gradient backpropagation algorithm to calculate the gradient of the loss function with respect to the neural network parameters and the importance matrix S, that is, and During the optimization process, the gradient backpropagation algorithm is combined with the optimization method to update the neural network f and the importance matrix S to minimize the loss function; S4. After m iterative training, the optimized neural network f* and feature importance matrix S* are obtained. Among them, the features encoded by f* are decoupled from each other, and the diagonal elements of S* represent the importance of the corresponding feature dimension. The larger the value of the diagonal element, the more important the corresponding dimension. The feature importance is ranked according to the size of the diagonal elements of S*.

2. The interpretable self-supervised pre-training method based on feature decoupling according to claim 1, characterized in that: The step S1 initializes the neural network f, and the operation includes: using the Python programming language to construct an 18-layer deep convolutional neural network, and using the deep learning toolkit to randomly initialize the parameter values of the neural network f.

3. The interpretable self-supervised pre-training method based on feature decoupling according to claim 1, characterized in that: The step S2 initializes the non-zero and non-negative diagonal elements in the importance matrix S, which are random numbers uniformly distributed in the interval (0, 1).

4. The interpretable self-supervised pre-training method based on feature decoupling according to claim 1, characterized in that: The number of training times in step S3 is determined based on the complexity of the problem and the available computing resources.

5. The interpretable self-supervised pre-training method based on feature decoupling according to claim 1, characterized in that: The data set in step S31 uses the CIFAR-10 data set, and the data enhancement operations include random cropping, scaling, flipping, color jittering, grayscale change and Gaussian blur processing.

6. The interpretable self-supervised pre-training method based on feature decoupling according to claim 1, characterized in that: In step S32, the data-enhanced samples are subjected to feature encoding by a neural network, and forward propagation of the network is used to obtain their feature representations.

7. The interpretable self-supervised pre-training method based on feature decoupling according to claim 1, characterized in that: In the process of calculating the loss function in step S34, the gradient clipping method is first used to avoid updating the non-diagonal elements of the importance matrix S, so that the non-diagonal elements remain 0. At the same time, an activation function Sigmoid is applied to the diagonal elements of the importance matrix S to keep the diagonal elements non-negative, and then the loss function calculation is entered.

8. The interpretable self-supervised pre-training method based on feature decoupling according to claim 1, characterized in that: The optimization method of step S35 uses the layered adaptive rate scaling algorithm LARS algorithm, and its setting parameters include the initial value of the learning rate, the learning rate update strategy and the weight decay parameter; the initial learning rate is set to 4.0, the cosine decay strategy is used to update the learning rate and the weight decay parameter is set to 0.000001.

Images