A semi-supervised iterative training method for remote sensing image interpretation in complex scenes

Abstract

The application discloses a kind of semi-supervised iterative training methods for complex scene remote sensing image interpretation, including the following steps: preliminary annotation is accumulated by semi-automatic acquisition or artificial labeling mode in early model training, training sample set is constructed to obtain annotation model by training, and interpretation model library is expanded;Cross pseudo-label constraint self-training is used to construct annotation model, and unlabeled data is gradually introduced to obtain more representative scene sample annotation, realize sample expansion and refinement.The application uses multi-modal network, based on the iterative training technology of semi-supervised learning such as self-training and cross pseudo-label, obtains the model set under large-area complex scene, based on scene rule set and model search, constructs model matching and integration technology, realizes large-area remote sensing image classification interpretation under complex scene;Through the connection of multi-modal data, sample library and model library, the continuous iteration of model and sample optimization is realized.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing technology, and in particular to a semi-supervised iterative training method for remote sensing image interpretation in complex scenes. Background Art

[0002] Applications in large-scale, complex scenarios face complex scenarios due to spatiotemporal variability. These challenges arise from complex meteorological conditions and diverse modal combinations across large regions. Furthermore, diverse topography and landforms across large regions make representative sample coverage difficult. Currently, research on intelligent remote sensing image classification based on multimodal remote sensing data is relatively limited, and applications in large-scale, complex scenarios are even rarer. This is primarily due to the challenges of multimodal data processing and sample annotation, as well as the diverse modal combinations within large-scale, complex scenarios. Furthermore, the complexity of these scenarios exacerbates the difficulty of model training.

[0003] Therefore, the classification and interpretation of large-area multimodal remote sensing data in complex scenarios faces challenges such as high sample preparation costs, complex and diverse scenarios, and insufficient generalization accuracy of models over large areas. At the same time, intelligent classification and interpretation require the construction of sophisticated models, which leads to problems such as small samples, large spatiotemporal differences in sample modalities, complex processes, and insufficient generalization performance in model training. It is necessary to construct targeted training and interpretation technologies for the intelligent classification of large-area multimodal remote sensing images in complex scenarios to solve problems such as the comprehensive application of multimodal data and small sample training in the intelligent classification of large-area images.

[0004] Semi-supervised training technology is a technique that reduces training costs and improves training effectiveness. Patent CN 113785314A discloses semi-supervised training of machine learning models using label guessing. Specifically, it discloses generating processed unlabeled batches and processed labeled batches from unlabeled batches and labeled batches, and using these data to train the machine learning model. This method uses the easy-to-use Mixmatch technology to perform data augmentation on the unlabeled data to generate K new data. Because data augmentation introduces noise, these K new data are input into the same classifier to obtain different predicted classification probabilities. By making these outputs self-consistent, the model converges. However, current applications of this method to complex multimodal remote sensing data face the following challenges:

[0005] (1) There is no semi-supervised training method suitable for complex scene multimodal remote sensing data;

[0006] (2) does not support multimodal data;

[0007] (3) The application of general semi-supervised methods in remote sensing multimodal training still needs to solve the problem of multimodal data segmentation;

[0008] (4) Lack of closed-loop iteration capabilities.

[0009] There is currently no effective solution to the above problems. Summary of the Invention

[0010] In response to the above-mentioned technical problems in related technologies, the present invention proposes a semi-supervised iterative training method for remote sensing image interpretation in complex scenes, which solves the problems of high sample preparation cost, complex and diverse scenes, and insufficient generalization accuracy of the model in large areas in the intelligent classification and interpretation of large-area remote sensing images in complex scenes, and can overcome the above-mentioned shortcomings of the existing technology.

[0011] To achieve the above technical objectives, the technical solution of the present invention is implemented as follows:

[0012] A semi-supervised iterative training method for remote sensing image interpretation in complex scenes includes the following steps:

[0013] S1 model training accumulates preliminary annotations through semi-automatic collection or manual annotation, builds a training sample set for training to obtain the annotation model, and expands the interpretation model library;

[0014] S2 builds a labeling model in a self-training manner with cross-pseudo-label constraints, and gradually introduces unlabeled data to obtain sample annotations for more representative scenarios, achieving sample expansion and refinement;

[0015] The specific steps of S2 are as follows:

[0016] S211 scores unlabeled samples, discards simple samples with high scores, and retains difficult samples with medium and low scores as candidate samples;

[0017] S212 introduces a batch of difficult samples into the training set for training. The samples whose prediction evaluation meets the confidence condition are retained, and the samples that do not meet the condition are selected as candidate samples.

[0018] S213 iterates step S212 until the stopping condition is met, and a more stable model is obtained;

[0019] For unlabeled data, S3 adopts the cross-pseudo-label supervision method to perform consistency constraint training. The same model is used but different initializations are used to generate difference models. Forward predictions are performed on the same input data to obtain probability results and pseudo-labels, and the pseudo-labels are used as supervision information of each other's models to calculate the loss and perform backward propagation. To obtain smoother training, a random mapping method is used to construct enhanced samples. Labeled samples are randomly attached to pseudo-label samples to generate new samples. The losses of the mapped samples are calculated separately for the real annotation and pseudo-label parts. During the training process, a dynamic adjustment method of loss normalization is used to obtain a more stable loss.

[0020] Furthermore, the specific steps of the self-training algorithm are as follows:

[0021] S221 uses the initial labeled data set as the initial training set and obtains a model based on the training set;

[0022] S222 uses the model to predict samples in the unlabeled dataset and selects the most reliable high-confidence samples;

[0023] S223 removes the high confidence samples selected in the previous step from the unlabeled samples. The high confidence samples here are pseudo-labeled samples.

[0024] S224 adds high confidence samples to the labeled dataset to obtain a new training set;

[0025] S225 trains a new model based on the new training set;

[0026] S226 repeats steps S222 to S225 until the stopping condition is met, that is, a stable annotation model is obtained.

[0027] Furthermore, the normalized loss in step S3 is expressed as:

[0028]

[0029] Among them, L h is the true label loss, L p is the pseudo label loss, is the sliding average of the true label loss, is the sliding average of the pseudo-label loss, and α is the weight coefficient.

[0030] Furthermore, the sliding average is calculated using an exponential sliding average, and the sliding decay rate is 0.9997.

[0031] The beneficial effects of the present invention are as follows: the present invention utilizes a multimodal network and iterative training techniques based on semi-supervised learning such as self-training and cross-pseudo-labeling to obtain a model set in large-area complex scenes, and constructs a model matching and integration technology based on a scene rule set and model search to realize the classification and interpretation of large-area remote sensing images in complex scenes; by connecting multimodal data, sample libraries and model libraries, continuous iteration of the model and sample optimization are realized, which greatly supports the construction of a multi-scene multimodal model set in complex scene large-area remote sensing interpretation. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0033] Figure 1 This is a technical flowchart of a semi-supervised iterative training method for remote sensing image interpretation in complex scenarios according to an embodiment of the present invention;

[0034] Figure 2 3 is a schematic diagram of cross-pseudo-label supervision of a semi-supervised iterative training method for remote sensing image interpretation in complex scenes according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention are within the scope of protection of the present invention.

[0036] like Figure 1-2 As shown, a semi-supervised iterative training method for remote sensing image interpretation in complex scenes according to an embodiment of the present invention includes the following steps:

[0037] S1 model training accumulates preliminary annotations through semi-automatic collection or manual annotation, builds a training sample set for training to obtain the annotation model, and expands the interpretation model library;

[0038] S2 builds a labeling model in a self-training manner with cross-pseudo-label constraints, and gradually introduces unlabeled data to obtain sample annotations for more representative scenarios, achieving sample expansion and refinement;

[0039] The specific steps of S2 are as follows:

[0040] S211 scores unlabeled samples, discards simple samples with high scores, and retains difficult samples with medium and low scores as candidate samples;

[0041] S212 introduces a batch of difficult samples into the training set for training. The samples whose prediction evaluation meets the confidence condition are retained, and the samples that do not meet the condition are selected as candidate samples.

[0042] S213 iterates step S212 until the stopping condition is met, and a more stable model is obtained;

[0043] S3 For unlabeled data, a cross-pseudo label supervision method is used for consistency constraint training, a same model is used but different initialization is used to generate different models, probability results and pseudo labels are obtained by respectively performing forward prediction on a same input data, and the pseudo labels are used as supervision information of each other model to calculate a loss and perform back propagation; in order to obtain smoother training, a random mapping method is used to construct enhanced samples, a sample with a label is randomly mapped to a pseudo label sample to generate a new sample, and the loss of the mapped sample with respect to the real label and the pseudo label part is respectively calculated, and a loss normalization dynamic adjustment method is used in the training process to obtain a more stable loss.

[0044] In the embodiment, the specific steps of the self-training algorithm are as follows:

[0045] S221 An initial labeled data set is used as an initial training set, and a model is obtained by training according to the training set;

[0046] S222 The model is used to predict samples in an unlabeled data set, and high-confidence samples with the most confidence are selected;

[0047] S223 The high-confidence samples selected in the previous step are removed from the unlabeled samples, and the high-confidence samples are pseudo label samples;

[0048] S224 The high-confidence samples are added to the labeled data set to obtain a new training set;

[0049] S225 A new model is trained according to the new training set;

[0050] S226 Steps S222 to S225 are repeated until a stop condition is met, that is, a stable labeling model is obtained.

[0051] In the embodiment, the normalized loss in step S3 is expressed as:

[0052]

[0053] wherein L h is a real label loss, L p is a pseudo label loss, is a sliding average of the real label loss, is a sliding average of the pseudo label loss, and a is a weight coefficient.

[0054] In the embodiment, the sliding average is calculated by using an exponential smoothing average, and the sliding decay rate is 0.9997.

[0055] In order to facilitate the understanding of the above technical solutions of the present application, the above technical solutions of the present application will be described in detail below through specific use modes.

[0056] In specific use, according to the semi-supervised iterative training method for remote sensing image interpretation in complex scenes described in the present invention, semi-supervised training techniques such as self-training and cross-pseudo-label supervision are used to achieve self-training on unlabeled data. By connecting multimodal data, sample libraries, and model libraries, continuous iteration and sample optimization of the model are formed, realizing an integrated "sample-training-prediction" iterative training technology. A semi-supervised iterative training technology route for remote sensing interpretation in complex scenes is shown in [1]. Figure 1 .

[0057] The connection between multimodal data and sample library is achieved through two methods. One is to manually label the multimodal remote sensing data, align and slice the labeled multimodal data, obtain sample data and update the sample library; the other is to align and slice the unlabeled multimodal remote sensing data, obtain the labeling through model prediction, and update the sample library.

[0058] The present invention uses a combination of self-training and semi-supervised training techniques with cross-pseudo-label supervision to achieve sample-to-model acquisition. Self-training is a semi-supervised training method that does not require assumptions. The algorithm flow of self-training is as follows:

[0059] 1) Use the initial labeled dataset as the initial training set and train a model based on the training set;

[0060] 2) Use the model to predict samples in the unlabeled dataset and select the most reliable high-confidence samples;

[0061] 3) Remove the high confidence samples selected in the previous step from the unlabeled samples, that is, the pseudo-label samples;

[0062] 4) Add high-confidence samples to the labeled dataset to obtain a new training set;

[0063] 5) Train a new model based on the new training set, repeat steps 2 to 5 until the stopping condition is met (for example, all unlabeled samples are labeled), and a stable model is obtained.

[0064] To support continuous optimization technology construction in complex scenarios, we build a labeling model in a self-training manner and gradually introduce unlabeled data to obtain more sample labels for representative scenarios, thereby achieving sample expansion and refinement. The process is as follows:

[0065] 1) Score the unlabeled samples, discard the simple samples with high scores, and retain the difficult samples with medium and low scores as candidate samples;

[0066] 2) A batch of difficult examples are introduced into the training set for training. The samples whose prediction evaluation meets the confidence condition are retained, and the samples that do not meet the condition are selected as candidate samples;

[0067] 3) Iterate step 2 until the stopping condition is met (for example, all unlabeled samples are labeled) to obtain a stable model.

[0068] For unlabeled data, this paper adopts a cross-pseudo-label supervision method to perform consistency constraint training. It mainly uses the same model but generates a difference model through different initialization. It performs forward prediction on the same input data to obtain probability results and pseudo-labels, and uses the pseudo-labels as supervision information of each model to calculate the loss and perform backpropagation. The process of the cross-pseudo-label supervision method is shown in Figure 2 .

[0069] To achieve smoother training, we use a random mapping method to construct augmented samples. We randomly map labeled samples to pseudo-labeled samples to generate new samples. The loss of the mapped samples is calculated separately for the true label and pseudo-label parts.

[0070] The loss used in standard self-training is the weighted sum of the true label loss and the pseudo label loss:

[0071] L=L h +αL p

[0072] Among them, L h , L p are the true label loss and pseudo label loss respectively, and α is the pseudo label loss weight coefficient.

[0073] However, this loss is not stable. The quality of pseudo-labeled data is not consistent at different stages of training. It is generally rough in the early stage and more accurate in the later stage. In order to obtain more stable training, a dynamic adjustment method of loss normalization is used during training to obtain a more stable loss. The normalized loss is expressed as:

[0074]

[0075] Among them, L h , L p 、 They are the true label loss, pseudo label loss and their respective sliding averages. The sliding average is calculated using exponential sliding average with a sliding decay rate of 0.9997. α is the weight coefficient.

[0076] The closed-loop iterative training process of the present invention is as follows: in the early stage of model training, preliminary annotations are accumulated through semi-automatic collection or manual annotation, a training sample set is constructed for training to obtain a annotation model, and an interpretation model library is expanded; in the middle stage, the annotation model is used as an interpretation model, application prediction is performed on multimodal remote sensing data to obtain pseudo labels, the pseudo labels are analyzed and mined to obtain wrong examples and difficult examples in the labels, the wrong examples and difficult examples are expanded to the sample library, an updated training sample set is obtained, iterative training is performed to obtain an updated annotation model and expand the interpretation model library, and continuous iterative training is performed using a training framework; in the later stage, the annotation model is used as an interpretation model, application prediction is performed on multimodal remote sensing data to obtain pseudo labels, the pseudo labels are analyzed and mined to obtain wrong examples and difficult examples in the labels, the wrong examples and difficult examples are expanded to the sample library, an updated training sample set is obtained, iterative training is performed to obtain a business model and expand it to the interpretation model library.

[0077] In summary, with the help of the above-mentioned technical solution of the present invention, a multimodal network is utilized, and iterative training techniques of semi-supervised learning such as self-training and cross-pseudo-labeling are used to obtain a model set for large-area complex scenes. Based on the scene rule set and model search, a model matching and integration technology is constructed to realize the classification and interpretation of large-area remote sensing images in complex scenes. By connecting multimodal data, sample library and model library, continuous iteration of the model and sample optimization are realized, which greatly supports the construction of a multi-scene multimodal model set in complex scene large-area remote sensing interpretation.

[0078] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A semi-supervised iterative training method for remote sensing image interpretation in complex scenes, characterized by: The steps include: S1 model training accumulates preliminary annotations through semi-automatic collection or manual annotation, builds a training sample set for training to obtain the annotation model, and expands the interpretation model library; S2 builds a labeling model in a self-training manner with cross-pseudo-label constraints, and gradually introduces unlabeled data to obtain sample annotations for more representative scenarios, achieving sample expansion and refinement; The specific steps of S2 are as follows: S211 scores unlabeled samples, discards simple samples with high scores, and retains difficult samples with medium and low scores as candidate samples; S212 introduces a batch of difficult samples into the training set for training. The samples whose prediction evaluation meets the confidence condition are retained, and the samples that do not meet the condition are selected as candidate samples. S213 iterates step S212 until the stopping condition is met, and a more stable model is obtained; For unlabeled data, S3 adopts the cross-pseudo-label supervision method to perform consistency constraint training. The same model is used but different initializations are used to generate difference models. Forward predictions are performed on the same input data to obtain probability results and pseudo-labels, and the pseudo-labels are used as supervision information of each other's models to calculate the loss and perform backward propagation. To obtain smoother training, a random mapping method is used to construct enhanced samples. Labeled samples are randomly attached to pseudo-label samples to generate new samples. The losses of the mapped samples are calculated separately for the real annotation and pseudo-label parts. During the training process, a dynamic adjustment method of loss normalization is used to obtain a more stable loss.

2. The semi-supervised iterative training method for remote sensing image interpretation in complex scenes according to claim 1, characterized in that: The specific steps of the self-training algorithm are as follows: S221 uses the initial labeled data set as the initial training set and obtains a model based on the training set; S222 uses the model to predict samples in the unlabeled dataset and selects the most reliable high-confidence samples; S223 removes the high confidence samples selected in the previous step from the unlabeled samples. The high confidence samples here are pseudo-labeled samples. S224 adds high confidence samples to the labeled dataset to obtain a new training set; S225 trains a new model based on the new training set; S226 repeats steps S222 to S225 until the stopping condition is met, that is, a stable annotation model is obtained.

3. The semi-supervised iterative training method for remote sensing image interpretation in complex scenes according to claim 1, characterized in that: The normalized loss in step S3 is expressed as: Among them, L h is the true label loss, L p is the pseudo label loss, is the sliding average of the true label loss, is the sliding average of the pseudo-label loss, and α is the weight coefficient.

4. The semi-supervised iterative training method for remote sensing image interpretation in complex scenes according to claim 3, characterized in that: The sliding average is calculated using an exponential sliding average, and the sliding decay rate is 0.9997.

Images