An efficient remote sensing image self-supervised learning training method

Abstract

The application discloses a self-supervised learning training acceleration method, which realizes evaluation of sample difficulty by extracting gradient normalization values of network loss, and then samples data with appropriate difficulty for training based on the difficulty score of the sample, accelerates convergence of the network under limited computing resources, and realizes more efficient self-supervised learning. Compared with a general self-supervised learning training method, the method disclosed by the application can save about 50% of training time under the same computing device, and the application can be effectively applied to a self-supervised learning task based on deep learning, effectively accelerates convergence, and saves computing resources.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and specifically relates to an efficient self-supervised learning training method for remote sensing images. Background Technology

[0002] Currently, my country has multiple remote sensing satellite systems in orbit, generating hundreds of terabytes of remote sensing image data daily. This vast amount of data provides a solid foundation for remote sensing image interpretation tasks. However, the time-consuming and labor-intensive process of labeling remote sensing images, coupled with the high cost of training with labeled data, has spurred the development of self-supervised learning for remote sensing images. Self-supervised learning aims to learn better feature representations of remote sensing images using unlabeled data, enabling these features to be transferred to downstream image interpretation tasks such as target detection, scene classification, and land cover classification, achieving improved performance.

[0003] By mining the inherent spatial or semantic associations of images, self-supervised learning can extract general representations from massive amounts of remote sensing data. However, due to the lack of label information, self-supervised training generally requires far more iterations than fully supervised training. Following a typical training process for self-supervised learning of remote sensing images, iterating for 200 epochs on a dataset of millions requires approximately 1600 cal-hours on a Tesla V100. This enormous computational resource requirement not only creates a barrier for small and medium-sized research institutions but also makes it difficult to iterate and optimize self-supervised models and methods. Therefore, it is necessary to optimize the training strategy for self-supervised remote sensing images, utilize computational resources more effectively, and design an efficient self-supervised learning training method for remote sensing images. Summary of the Invention

[0004] Experiments revealed that self-supervised learning of remote sensing images is computationally intensive, and treating every training sample equally is inefficient. Therefore, we optimized the training process by adjusting the difficulty of the training samples. In remote sensing images, targets of interest often exhibit clustered distributions. Some areas, such as cities, have a large number of targets of interest in their corresponding remote sensing images, making them difficult to extract data from. In contrast, areas like mountains, fields, and sea surfaces contain fewer targets of interest with similar appearances, making the samples relatively simple. Current algorithms treat all training data equally, leading to low training efficiency. In the early stages of network training, the network cannot handle overly difficult data, and the gradients generated by such data tend to be close to noise, causing network instability or even gradient explosion. In the later stages of network training, simpler data is not very helpful, and excessively small gradients dilute the training effect of other samples. Therefore, assigning appropriate sampling weights to different types of samples at appropriate training stages can help improve the network's training efficiency.

[0005] However, in self-supervised learning of remote sensing images, using only the magnitude of the loss generated by the network to measure the difficulty of the training samples is rather one-sided. On the one hand, the loss generated by the network can only reflect the strength of the model's representation ability at the current moment, and cannot reflect the semantic richness or training value of the data itself. On the other hand, self-supervised learning uses highly intensive data augmentation strategies, so the loss generated by the network cannot directly represent the difficulty of the image itself, and will be affected by the data augmentation.

[0006] To address the aforementioned technical problems, this invention assesses sample difficulty by extracting gradient normalization (GradNorm) from the network loss and designs an optimized training strategy based on sample difficulty, enabling more efficient self-supervised learning training. The technical solution adopted in this invention is: an efficient self-supervised learning training method for remote sensing images, specifically comprising:

[0007] A self-supervised network for remote sensing images is constructed and trained to extract features from remote sensing images. This self-supervised network consists of a student-teacher network, where the student network contains a feature extractor f. θ Feature projector g θ Feature predictor q θ The teacher network includes a feature extractor f ξ and feature projector g ξ ; Pairs of remote sensing image views with different data augmentations v θ and v′ ξ Input the student network and the teacher network respectively. In both networks, the feature extractor is used to extract the depth feature map of the view to obtain y. θ y′ ξ The feature projector, on the other hand, transforms the extracted feature map into a feature projection vector, thus obtaining z. θ and z′ ξ The feature predictor of the student network predicts the feature projection vector q of the student network. θ (z θ Predict the feature projection vector z′ of the teacher network. ξ The loss function is the mean squared error between the predicted vector and the feature projection vector of the teacher network. During iterative training, samples are selected for training based on sample difficulty. The student network updates its parameters according to backpropagation, and the teacher network's parameters are the exponential moving average of the student network's parameters.

[0008] Specifically, the training process involves the following steps:

[0009] Step 01: The first generation of iterative training utilizes all training data. Based on the gradient normalization value of the self-supervised loss for each sample during training, a difficulty score is assigned to each training sample. Specifically, the sample difficulty score is obtained by calculating the gradient of the last fully connected layer in the feature predictor of the student network. This layer is directly related to the sample features and sample loss, and is located in a shallow layer on the backpropagation path, making it less susceptible to gradient vanishing.

[0010] Step 02: Iterative training from the second to the Kth generation (empirical value of 10). Based on the difficulty scores of the previous generation, select data from the training set whose difficulty distribution is between the 50% and 70% quantiles. All data in this range will be used for training, accounting for 20% of the total data. In addition, 5% of the data from the 0% to 50% quantile and the 70% to 100% quantiles will be randomly sampled and used for training, for a total of 30% of the total data.

[0011] Step 03: During each round of sampling, the difficulty score of the samples participating in training also needs to be calculated, and the difficulty score of all samples is updated during the next generation of training.

[0012] Step 04: Repeat steps 01 to 03 for every K generations of training.

[0013] Furthermore, the loss function in the training process of the self-supervised network for remote sensing images is the mean square error of the predicted vector and the normalized feature projection vector of the teacher network, i.e. in

[0014] Furthermore, sample difficulty score The calculation method is as follows, where L is the total number of samples in the training set. Let z be the feature projection vector output by the student network. θ The feature projection vector output by the teacher network is z′ ξ ;

[0015] The last fully connected layer before the student network output is located in the feature predictor q. θ In the above, we assume the feature predictor q θ It has a total of n fully connected layers The output feature of the (n-1)th layer is M is the dimension of feature u, circle If the symbol represents nested functions, then the characteristic of the output of the nth level is: N is the dimension of feature v; the learning objective of the self-supervised network for remote sensing images is to improve the output of the student network. and the output of the teacher network With a closer distance, the loss function is:

[0016]

[0017] set up The weight parameters are Then v can be written as loss function about The gradient is:

[0018]

[0019] The transpose of this expression is determined by factors of the chain rule. This is generated to ensure that the loss function is related to... The gradient is an N×M matrix, not a 1×NM row vector; for the first term on the right:

[0020]

[0021] For the second term on the right, the normalization operation... The gradient is an N×N Jacobian matrix. The partial derivative of the j-th output element with respect to the i-th input element can be expressed as:

[0022]

[0023] Among them, v i v i v k δ represents the value of vector v in the i, j, k dimensions. ij The symbol for Kronecker.

[0024] δ ij =1, δ in other cases ij =0; therefore, the overall Jacobian matrix of the second term is:

[0025]

[0026] I∈R N×N Since v is the identity matrix of the same order as v, formula (2) can be expressed as:

[0027]

[0028] Each element of G:

[0029]

[0030] The gradient normalization result generated for each sample is as follows:

[0031]

[0032] In the formula All are unit vectors. Let cosine be the angle between the two vectors. represent exist Projection onto the surface; ignoring the scaling factor 2 / ||v||, The received gradient is proportional to the L1 norm of the hidden layer feature u, and proportional to the output of the student network. In the output of the teacher network Projection differences The L1 norm of cosψ is proportional to it.

[0033] Compared with existing technologies, the advantages and beneficial effects of this invention are as follows: This invention discloses an efficient self-supervised learning training method for remote sensing images. During the training process, the difficulty of samples is evaluated by extracting the gradient normalization value of the network loss. Then, based on the difficulty score of the samples, appropriately difficult data is sampled for training. This accelerates network convergence with limited computing resources, achieving more efficient self-supervised learning. Compared with general self-supervised learning training methods, under the same computing equipment, the method disclosed in this invention can save approximately 50% of the training time. This invention can be effectively applied to deep learning-based self-supervised learning tasks, effectively accelerating convergence and saving computing resources. Attached Figure Description

[0034] Figure 1 This is a network framework diagram of an embodiment of the present invention;

[0035] Figure 2 This is a schematic diagram of the network training process according to an embodiment of the present invention;

[0036] Figure 3 This is a schematic diagram of the sample difficulty score calculation network structure in an embodiment of the present invention. Detailed Implementation

[0037] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0038] like Figure 1 As shown, the self-supervised network for remote sensing images consists of a student-teacher network, where the student network contains a feature extractor f. θ Feature projector g θ Feature predictor q θ The teacher network includes a feature extractor f ξ and feature projector g ξ Pairs of remote sensing image views will be augmented with different data enhancements. θ and v′ ξInput the student network and the teacher network respectively. In both networks, the feature extractor is used to extract the depth feature map of the view to obtain y. θ y′ ξ The feature projector, on the other hand, transforms the extracted feature map into a feature projection vector, thus obtaining z. θ and z′ ξ The feature predictor of the student network is based on the feature projection vector q of the student network. θ (z θ Predict the feature projection vector z′ of the teacher network. ξ The loss function is the mean squared error of the predicted vector and the normalized feature projection vector of the teacher network, i.e. in During iterative training, samples are selected for training based on sample difficulty to train the network. The student network updates its parameters through backpropagation, and the teacher network's parameters are an exponential moving average of the student network's parameters.

[0039] The training process of the network is as follows Figure 2 As shown, it mainly includes the following steps:

[0040] Step 01: The first generation of iterative training utilizes all training data. Based on the gradient normalization value of the self-supervised loss for each sample during training, a difficulty score is assigned to each training sample. Specifically, the sample difficulty score is obtained by calculating the gradient of the last fully connected layer in the feature predictor of the student network. This layer is directly related to the sample features and sample loss, and is located in a shallow layer on the backpropagation path, making it less susceptible to gradient vanishing.

[0041] Step 02: Iterative training from the second to the Kth generation (empirical value of 10), sampling 30% of all data for training. First, based on the difficulty scores of the previous generation, select data whose difficulty distribution in the training set falls between the 50% and 70% quantiles. All data in this range are used for training, accounting for 20% of the total data. Additionally, 5% of the data from the 0% to 50% quantile and the 70% to 100% quantiles are randomly sampled and used for training, resulting in a total of 30% of the data participating in training. These values ​​are the optimal values ​​obtained from experiments; other values ​​can also be used.

[0042] Step 03: During each round of sampling, the difficulty score of the samples participating in training also needs to be calculated, and the difficulty score of all samples is updated during the next generation of training.

[0043] Step 04: Repeat steps 01 to 03 for every K generations of training.

[0044] Specifically, sample difficulty score The calculation method is as follows, where L is the total number of samples in the training set. Let z be the feature projection vector output by the student network. θ The feature projection vector output by the teacher network is z′ ξ The sample difficulty score calculation process is as follows:

[0045] like Figure 3 As shown, the last fully connected layer before the student network output is located in the feature predictor q. θ Without loss of generality, we assume that the predictor q θ It has a total of n fully connected layers The output feature of the (n-1)th layer is M is the dimension of feature u, circle Symbols indicating nested functions Then the characteristics of the output of the nth layer are N is the dimension of feature v. The learning objective of this network is to improve the output of the student network. and the output of the teacher network With a closer distance, the loss function is defined as:

[0046]

[0047] set up The weight parameters are Then v can be written as loss function about The gradient is:

[0048]

[0049] Note that the transpose of this expression is influenced by factors of the chain rule. This is generated to ensure that the loss function is related to... The gradient is an N×M matrix, not a 1×NM row vector. For the first term on the right:

[0050]

[0051] For the second term on the right, the normalization operation... The gradient is an N×N Jacobian matrix. The partial derivative of the j-th output element with respect to the i-th input element can be expressed as:

[0052]

[0053] Among them, v i v i v k δ represents the value of vector v in the i, j, k dimensions. ij The symbol for Kronecker.

[0054] δ ij =1, δ in other cases ij =0. Therefore, the overall Jacobian matrix of the second term is:

[0055]

[0056] I∈R N×N Let v be the identity matrix of the same order as v. Therefore, formula (2) can be expressed as:

[0057]

[0058] Each element of G:

[0059]

[0060] The gradient normalization result generated for each sample is as follows:

[0061]

[0062] In the formula All are unit vectors. Let cosine be the angle between the two vectors. represent exist The projection onto the surface. Ignoring the scaling factor 2 / ||v||, The received gradient is proportional to the L1 norm of the hidden layer feature u, and proportional to the output of the student network. In the output of the teacher network Projection differences The gradient is proportional to the L1 norm. Therefore, the above equation has a clear physical meaning: the higher the activation level of the hidden layer feature u, i.e., the richer the semantics, the larger the gradient generated by the network; the greater the difference between the outputs of the student network and the teacher network, the larger the gradient generated by the network will also be. Thus, a score representing the sample difficulty can be obtained.

[0063] By employing the gradient-based sampling strategy designed in this invention, the network can receive samples of moderate difficulty at each stage. These samples achieve a balance between semantic richness and learning difficulty, enabling the network to achieve faster convergence speed with limited computing resources. Experiments on ImageNet have verified that the method of this invention can achieve the same training effect while saving approximately 50% of the training time.

[0064] In summary, the efficient self-supervised learning training method for remote sensing images described in this invention mainly assesses the difficulty of samples by extracting and normalizing the gradient of the network loss. Then, based on the sample difficulty, it selects appropriate data for training, achieving faster convergence speed and more efficient self-supervised learning with limited computational resources. Final experiments verify that the method described in this invention can save approximately 50% of training time. This invention can be effectively applied to deep learning-based self-supervised learning tasks, effectively accelerating convergence and saving computational resources.

[0065] It should be understood that any parts not described in detail in this specification belong to the prior art.

[0066] It should be understood that the above description of the preferred embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art, under the guidance of this invention, can make substitutions or modifications without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. An efficient self-supervised learning training method for remote sensing images, characterized in that, Specifically, it includes: A self-supervised network for remote sensing images is constructed and trained to extract features from remote sensing images. The self-supervised network consists of a student-teacher network, where the student network contains a feature extractor. Feature projector Feature predictor The teacher network includes a feature extractor. and feature projector Pairs of remote sensing image views with different data augmentations and Inputting data into both the student and teacher networks, respectively, allows feature extractors in both networks to extract depth feature maps of the views. , The feature projector, on the other hand, transforms the extracted feature map into a feature projection vector, thus obtaining... and The feature predictor of the student network is based on the feature projection vector of the student network. Predicting the feature projection vectors of the teacher network During iterative training, the training network is trained based on the difficulty of the training samples. The student network updates its parameters according to backpropagation, and the parameters of the teacher network are the exponential moving average of the student network parameters. Specifically, the training process involves the following steps: Step 01: The first generation of iterative training uses all training samples for training. Based on the gradient normalization value of the self-supervised loss of each training sample during the training process, the difficulty of each training sample is scored. Step 02: Sampling the training samples: Iterative training from the second generation to the [missing information - likely a specific generation or stage]. In the next generation, based on the sample difficulty scores of the previous generation, data with difficulty distribution between the M1% and M2% quantiles of the training set are selected. All data in this range are used for training, accounting for (M2-M1)% of the total data. In addition, 5% of the data from the 0% to M1% quantile and the M2% to 100% quantile are randomly sampled and used for training. That is, a total of (M2-M1+5×2)% of the total data are used for training. Step 03: During each round of sampling, the difficulty score of the samples participating in training also needs to be calculated, and the difficulty score of all samples is updated during the next generation of training. Step 04: Each training session Repeat steps 01 to 03. The loss function in the training process of a self-supervised network for remote sensing images is the mean square error of the predicted vector and the normalized feature projection vector of the teacher network, i.e. , in , ; The sample difficulty score is obtained by calculating the gradient of the last fully connected layer in the feature predictor of the student network. This layer is directly related to the sample features and sample loss, and is located in a shallow layer on the backpropagation path, making it less susceptible to gradient vanishing. Sample Difficulty Score The calculation method is as follows, where Let be the total number of samples in the training set, and denote the feature projection vector of the student network output as . The feature projection vector output by the teacher network is ; The last fully connected layer before the student network output is located in the feature predictor. In the context of feature predictors, the following is assumed: Together with A fully connected layer , No. The output features of the layer are , Features The dimension of the circle The symbol representing nested functions, then the first... The characteristics of the layer output are , Features The dimension; the learning objective of remote sensing image self-supervised networks is to improve the output of student networks. and the output of the teacher network With a closer distance, the loss function is defined as: set up The weight parameters are ,but Recorded as loss function about The gradient is: The transpose of this expression is determined by factors of the chain rule. This is generated to ensure that the loss function is related to... The gradient is a The matrix instead The row vector; for the first term on the right: For the second term on the right, the normalization operation... The gradient is a Jacobian matrix , No. The output element is the first... The partial derivatives of the input elements are expressed as: in, , , Representing vectors In the Dimension value, The symbol for Kronecker is used if and only if hour, In other cases Therefore, the overall Jacobian matrix for the second term is: To and The identity matrices are of the same order; therefore, formula (2) can be expressed as: in Each element: The gradient normalization result generated for each sample is as follows: In the formula All are unit vectors. Let cosine be the angle between the two vectors. represent exist Projection onto the surface; ignore scale factor back, Received gradients and hidden layer features of The norm is proportional to the student's online output. In the output of the teacher network Projection differences of The norm is directly proportional.

2. The efficient self-supervised learning training method for remote sensing images as described in claim 1, characterized in that: M1 is set to 50, and M2 is set to 70.

Images