Remote sensing multi-task interpretation method and device based on cross-knowledge interactive learning and self-supervised auxiliary learning

Abstract

The application discloses a remote sensing multi-task interpretation method and device based on cross-knowledge interactive learning and self-supervision auxiliary learning, relates to the technical field of remote sensing visual interpretation, and comprises the following steps: acquiring an image to be processed, setting task requirements of the image to be processed, the task requirements comprising one or more of a single-mode data classification task, a multi-mode data classification task, a single-mode data segmentation task, a multi-mode data segmentation task, a single-mode data change detection task and a multi-mode data change detection task; inputting the image to be processed into a trained multi-task interpretation network, and obtaining a result of the image to be processed after executing the task requirements according to the task requirements of the image to be processed; wherein the trained multi-task interpretation network is trained in a self-supervision learning mode by taking data of preset categories as a training data set, and is obtained by training an initial multi-task interpretation network. The application can realize the interpretation of remote sensing visual single-mode classification, segmentation, change detection and multi-mode classification, segmentation and change detection, and obtain high-precision interpretation results.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing visual interpretation technology, specifically relating to a remote sensing multi-task interpretation method and apparatus that combines cross-knowledge interactive learning and self-supervised assisted learning. Background Technology

[0002] Currently, multi-task learning, as an advanced machine learning paradigm, is demonstrating remarkable potential in multiple fields due to its unique advantages. The core idea of ​​multi-task learning lies in optimizing multiple related tasks in parallel, leveraging the complementarity between tasks, implementing parameter sharing strategies in shared network structure layers, and carefully constructing joint loss functions to enhance the model's generalization ability. This ensures balanced performance improvement across multiple tasks or achieves superior performance on specific tasks by utilizing auxiliary information from other tasks. Multi-task learning has made significant progress in the field of natural image processing. However, in multi-task learning, different tasks often use different datasets, and the differences between these datasets can impact model performance. In remote sensing images, due to the diversity of dataset modalities, there are often significant differences between datasets, which can greatly affect model performance. To realize a multi-task interpretation system for remote sensing, it is necessary to consider the differences in datasets and to balance the effects of different tasks. Existing technologies are still insufficient in addressing these challenges in multi-task learning of remote sensing images.

[0003] In existing technologies, Liu Shangwang et al. proposed a semantic segmentation method for buildings and features in remote sensing images based on multi-task learning of the Unet network. This network can improve the accuracy of semantic segmentation of buildings and features in remote sensing images and has a certain degree of accuracy and timeliness. However, the depth of the proposed network is still limited, and the boundary distance uses simple Euclidean distance. Ma Hui et al. proposed a multi-task learning deep Siamese network model, which is suitable for change detection scenarios with historical thematic maps but no changes or unchanged samples. However, the existing change detection accuracy does not meet the needs of practical engineering applications and needs to be combined with manual interpretation.

[0004] Therefore, there is an urgent need to provide a remote sensing visual multi-task interpretation method to address the aforementioned deficiencies in existing technologies. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, this invention provides a remote sensing multi-task interpretation method and apparatus based on cross-knowledge interactive learning and self-supervised assisted learning. The technical problem to be solved by this invention is achieved through the following technical solution:

[0006] In a first aspect, the present invention provides a remote sensing multi-task interpretation method that integrates cross-knowledge interactive learning and self-supervised assisted learning, comprising:

[0007] Acquire the image to be processed and set the task requirements for the image to be processed. The task requirements include one or more of the following: single-modal data classification task, multi-modal data classification task, single-modal data segmentation task, multi-modal data segmentation task, single-modal data change detection task, and multi-modal data change detection task.

[0008] The image to be processed is input into the trained multi-task interpretation network, and the result of the image after the task requirements are met is obtained according to the task requirements of the image to be processed.

[0009] The pre-trained multi-task interpretation network is trained using data of a preset category as the training dataset and trained in a self-supervised learning manner.

[0010] Secondly, the present invention also provides a remote sensing multi-task interpretation device that integrates cross-knowledge interactive learning and self-supervised assisted learning, comprising:

[0011] The image acquisition module is used to acquire the image to be processed and set the task requirements for the image to be processed. The task requirements include one or more of the following: single-modal data classification task, multi-modal data classification task, single-modal data segmentation task, multi-modal data segmentation task, single-modal data change detection task, and multi-modal data change detection task.

[0012] The image processing module is used to input the image to be processed into the trained multi-task interpretation network, and obtain the result of the image after the task requirements are met.

[0013] The pre-trained multi-task interpretation network is trained using data of a preset category as the training dataset and trained in a self-supervised learning manner.

[0014] The beneficial effects of this invention are:

[0015] This invention provides a remote sensing multi-task interpretation method and apparatus based on cross-knowledge interactive learning and self-supervised assisted learning. Addressing the issues of performance impact caused by the diversity of dataset modalities and insufficient multi-task expansion in current remote sensing visual multi-task technologies, this invention aims to reduce the impact of dataset modal differences by merging datasets from multiple tasks into a self-supervised learning network and utilizing self-supervised assisted learning. Furthermore, it innovatively proposes encoder partial parameter sharing and a joint loss function for seven tasks to achieve cross-knowledge interactive learning among multiple tasks.

[0016] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0017] Figure 1This is a flowchart of a remote sensing multi-task interpretation method based on cross-knowledge interactive learning and self-supervised assisted learning provided in an embodiment of the present invention;

[0018] Figure 2 This is a schematic diagram of a multi-task interpretation network provided in an embodiment of the present invention;

[0019] Figure 3 This is a schematic diagram of a first neck network and a second neck network provided in an embodiment of the present invention;

[0020] Figure 4 This is a schematic diagram of an encoder provided in an embodiment of the present invention. Detailed Implementation

[0021] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0022] Please see Figure 1 , Figure 1 This is a flowchart of a remote sensing multi-task interpretation method based on cross-knowledge interactive learning and self-supervised assisted learning provided in an embodiment of the present invention. The remote sensing multi-task interpretation method based on cross-knowledge interactive learning and self-supervised assisted learning provided by the present invention includes:

[0023] S101. Obtain the image to be processed and set the task requirements for the image to be processed. The task requirements include one or more of the following: single-modal data classification task, multi-modal data classification task, single-modal data segmentation task, multi-modal data segmentation task, single-modal data change detection task, and multi-modal data change detection task.

[0024] Specifically, in this embodiment, an image to be processed is acquired, and according to the task requirements, one or more of the following are performed on the image to be processed: classification, segmentation, or change detection.

[0025] S102. Input the image to be processed into the trained multi-task interpretation network, and obtain the result of the image after performing the task requirements according to the task requirements of the image to be processed.

[0026] The pre-trained multi-task interpretation network is trained using data of a preset category as the training dataset and trained in a self-supervised learning manner.

[0027] Specifically, please see Figure 2 , Figure 2 This is a schematic diagram of a multi-task interpreter network provided in an embodiment of the present invention. In this embodiment, before training the initial multi-task interpreter network, the following steps are also included:

[0028] Construct a multi-task interpretation network;

[0029] The multi-task interpretation network includes an encoder, a classifier, a classification fusion module, a first neck network, a first feature fusion module, a second neck network, a second feature fusion module, a first shared decoder, a second shared decoder, a third shared decoder, a fourth shared decoder, and a decoder. The encoder's input receives the data to be processed. The encoder's output is connected to the input of the classifier, the classification fusion module, the first neck network, the first feature fusion module, the second neck network, and the second feature fusion module. The encoder's output is connected to the decoder's input, and the decoder outputs the reconstructed data to be processed. The first neck network's output is connected to the first shared decoder's input, and the first shared decoder outputs the single-modal data segmentation result. The first feature fusion module's output is connected to the second shared decoder's input, and the second shared decoder outputs the multi-modal data segmentation result. The second neck network's output is connected to the third shared decoder's input, and the third shared decoder outputs the single-modal data change detection result. The second feature fusion module's output is connected to the fourth shared decoder's input, and the fourth shared decoder outputs the multi-modal data change detection result.

[0030] In this embodiment, please refer to Figure 3 , Figure 3 This is a schematic diagram of a first neck network and a second neck network provided in an embodiment of the present invention. The classification fusion module includes multiple classifiers; the first neck network and the second neck network include a feature pyramid network.

[0031] It should be noted that Feature Pyramid Networks have been widely used in tasks such as object detection and semantic segmentation, such as... Figure 3 As shown, the input image is processed through a backbone network to extract features, resulting in a series of feature maps at different scales. Feature fusion is then performed between different feature layers to obtain richer feature representations. The fused feature maps are then used for tasks such as object detection and semantic segmentation.

[0032] In this embodiment, please continue to refer to Figure 2 The trained multi-task interpretation network is obtained by training the initial multi-task interpretation network using data of a preset category as the training dataset and through self-supervised learning. This includes:

[0033] Obtain the training dataset, which includes a unimodal classification dataset, a multimodal classification dataset, a unimodal segmentation dataset, a multimodal segmentation dataset, a unimodal change detection dataset, a multimodal change detection dataset, and a task dataset;

[0034] Obtain the true labels corresponding to the training samples in the training dataset;

[0035] The training dataset and the real labels corresponding to the training samples in the training dataset are input into the initial multi-task interpretation network for iterative training. The loss function is minimized within a preset number of iterations, resulting in the trained multi-task interpretation network.

[0036] It should be noted that in this embodiment, the AdamW optimizer is used, with the learning rate set to 0.00006 and the weight decay set to 0.01 for training. After obtaining the optimal epoch, the results are tested.

[0037] In this embodiment, the method further includes: organizing the information of the datasets for different tasks and preprocessing the datasets to obtain datasets that meet the requirements.

[0038] Specifically, the information organization for the dataset mainly includes the dataset's modality, image size, and number of channels. To enable parameter sharing in subsequent experiments, it is necessary to ensure that the size and number of channels of the input dataset to the network are consistent.

[0039] In this example, all data is processed into a 512x512 three-channel dataset, and all datasets are scaled to 512x512 size using bilinear interpolation.

[0040] For SAR modes, consider two common types of data: one is a single-channel dataset, which is transformed into three-channel data through replication; the other is a two-channel dataset consisting of vv and vh, with the ratio of channel one to channel two used as the third channel to form a three-channel dataset.

[0041] For multispectral modes, three channels, R, G, and B, are selected to form a three-channel dataset.

[0042] The processed datasets from the six main tasks are merged together to form the dataset for the auxiliary tasks.

[0043] In this embodiment, please refer to Figure 4 , Figure 4 This is a schematic diagram of an encoder provided in an embodiment of the present invention. The training dataset and the real labels corresponding to the training samples in the training dataset are input into an initial multi-task interpretation network for iterative training, including:

[0044] The unimodal classification dataset is input into the encoder for processing to obtain the first feature. The first feature is then input into the classifier for normalization. The normalized first feature is mapped to the class space to obtain the predicted score of the training samples of each class in the unimodal classification dataset. The class probability distribution is generated through the activation function. The loss of the unimodal data classification task is obtained by comparing the predicted score of the training samples of each class in the unimodal classification dataset with the true label corresponding to the training samples in the unimodal classification dataset.

[0045] The single-modal segmentation dataset is input into the encoder for processing to obtain multi-layer second features. These multi-layer second features are then input into the first neck network for feature fusion. The fused features are then input into the first shared decoder for multi-scale feature processing to obtain the single-modal data segmentation map. The loss for the single-modal data segmentation task is obtained by comparing the single-modal data segmentation map with the ground truth labels corresponding to the training samples in the single-modal segmentation dataset. Optionally, the first neck network is used to fuse multi-layer features from the encoder, specifically fusing the 2nd, 5th, 8th, and 11th layers of ViT-Base, or the 7th, 11th, 15th, and 23rd layers of ViT-Large. It is understood that the obtained single-modal data segmentation map has the same size as the input image.

[0046] The single-modal change detection dataset is input into the encoder for processing to obtain multi-layer third features. These multi-layer third features are then input into the first feature fusion module for feature fusion to obtain fused features. The multi-layer third features are then input into the decoder for processing to obtain decoded features. The absolute difference between the fused features and the decoded features is calculated to obtain the difference features between the fused features and the decoded features. These difference features are then input into the second shared decoder for processing to obtain the result of the single-modal data change detection. The loss of the single-modal data change detection task is obtained by comparing the result of the single-modal data change detection with the true labels corresponding to the training samples in the single-modal change detection dataset. Optionally, the first feature fusion network is used to fuse the multi-layer features of the encoder, specifically fusing the 2nd, 5th, 8th, and 11th layers of features from ViT-Base, or fusing the 7th, 11th, 15th, and 23rd layers of features from ViT-Large.

[0047] The multimodal classification dataset is input into the encoder for processing to obtain multiple first features. These first features are then input into different classifiers for processing to obtain multiple predicted scores for the training samples of each category in the multimodal classification dataset. The average of these multiple predicted scores is then calculated. By comparing the average score of the training samples of each category in the multimodal classification dataset with the true labels corresponding to the training samples in the multimodal classification dataset, the loss of the multimodal data classification task is obtained.

[0048] The multimodal segmentation dataset is input into the encoder for processing to obtain multi-layer second features. These multi-layer second features are then input into the second neck network for feature fusion, resulting in multiple fused features. The average of these fused features is then input into the third shared decoder for multi-scale feature processing, yielding a multimodal data segmentation map. The loss for the multimodal data segmentation task is obtained by comparing the multimodal data segmentation map with the ground truth labels corresponding to the training samples in the multimodal segmentation dataset. Optionally, the second neck network is used to fuse multi-layer features from the encoder, specifically fusing features from layers 2, 5, 8, and 11 of ViT-Base, or features from layers 7, 11, 15, and 23 of ViT-Large. It is understood that the obtained multimodal data segmentation map has the same size as the input image.

[0049] The multimodal change detection dataset is input into the encoder for processing to obtain multi-layer third features. These multi-layer third features are then input into the second feature fusion module for feature fusion, resulting in multiple fused features. These multi-layer third features are then input into the decoder for processing, resulting in multiple decoded features. The absolute difference between the fused features and the decoded features is calculated to obtain multiple difference features between the fused features and the decoded features. The average of these difference features is then input into the fourth shared decoder for processing, yielding the multimodal data change detection result. The loss of the multimodal data change detection task is obtained by comparing the multimodal data change detection result with the ground truth labels corresponding to the training samples in the multimodal change detection dataset. Optionally, the first feature fusion network is used to fuse the multi-layer features of the encoder, specifically fusing the 2nd, 5th, 8th, and 11th layers of ViT-Base, or fusing the 7th, 11th, 15th, and 23rd layers of ViT-Large, such as... Figure 4 As shown, d = layers, counting from 0. It can be understood that the inputs to the decoder and the second neck network yield four features: features of mode 1 before the change, features of mode 2 before the change, features of mode 1 after the change, and features of mode 2 after the change. These are then used as inputs to the decoder for the following four features: the absolute difference between the features of mode 2 after the change and the features of mode 1 before the change, the absolute difference between the features of mode 2 after the change and the features of mode 2 before the change, the absolute difference between the features of mode 1 after the change and the features of mode 1 before the change, and the absolute difference between the features of mode 1 after the change and the features of mode 2 before the change. The average of these four features is then taken to obtain the difference feature.

[0050] In this embodiment, the training dataset and the real labels corresponding to the training samples in the training dataset are input into the initial multi-task interpretation network for iterative training, including:

[0051] The task dataset is input into the encoder for masking with a masking rate of 75%. The processed features are then input into the decoder for feature extraction to obtain the reconstructed task dataset. The self-supervised loss is obtained by comparing the features of the training samples in the task dataset with the features of the training samples in the reconstructed task dataset.

[0052] In this embodiment, the encoder processing implements the underlying shared parameter network, and the encoder processing includes:

[0053] The encoder divides the input data to be processed into multiple fixed-size image blocks and flattens each image block; it then embeds position information into the flattened image blocks.

[0054] Image patches with embedded location information are processed by cascaded coding layers in the encoder to obtain long-distance dependencies and contextual information in the image patches, resulting in multi-feature image patches. These multi-feature image patches are shared in single-modal data classification tasks, single-modal data segmentation tasks, single-modal data change detection tasks, multi-modal data classification tasks, multi-modal data segmentation tasks, and multi-modal data change detection tasks.

[0055] Understandably, in this embodiment, to fully utilize the potential correlations between multiple main task datasets and reduce performance bottlenecks caused by differences in different modal datasets, an innovative strategy is proposed to comprehensively integrate all main task datasets to construct a unified, diversified dataset. Based on this integrated dataset, an auxiliary self-supervised learning task is designed and implemented, aiming to mine deep structures and intrinsic patterns in the data in an unsupervised manner. In this framework, an encoder weight sharing mechanism is adopted. By sharing encoder weights across tasks, not only can the transfer and fusion of knowledge between different tasks be promoted, but model parameters can also be effectively reduced, improving training efficiency. More importantly, this weight sharing method enables the model to learn a more general and versatile data representation that can transcend the boundaries of different modalities and domains, providing strong support for subsequent main task learning.

[0056] In this embodiment, the first shared decoder shares network parameters with the second shared decoder, and the third shared decoder shares network parameters with the fourth shared decoder, to achieve high-level network parameter sharing. For single-modal image segmentation and multi-modal image segmentation, the network parameters of the shared decoder UPerNet are used; for single-modal change detection and multi-modal change detection, the network parameters of the decoder UPerNet are also used.

[0057] Understandably, in this embodiment, the same decoder, UPerNet, is used to handle both unimodal and multimodal tasks. This significantly reduces the total number of model parameters, accelerates the training process, and lowers the model's computational resource requirements, thus reducing computational costs during training. This is particularly important for large datasets and complex networks. Sharing the decoder across different tasks allows the model to learn general feature representations across tasks and modalities during training. This knowledge transfer helps improve the model's generalization ability on unseen data or new tasks.

[0058] In this embodiment, the loss function includes the loss L1 for single-modal data classification, the loss L2 for single-modal data segmentation, the loss L3 for single-modal data change detection, the loss L4 for multi-modal data classification, the loss L5 for multi-modal data segmentation, the loss L6 for multi-modal data change detection, and the self-supervised loss L... m The loss function L is expressed as:

[0059]

[0060] Where i∈[1,2,3,4,5,6], L i0 L represents the loss of the first iteration, ω and θ represent the learnable parameters. j and L i Both represent the loss during the iterative process.

[0061] Understandably, considering the rate of decrease in loss for different tasks, tasks with slower decrease rates should have their weights increased in the next iteration. Simultaneously, the current loss for different tasks should also be considered, with tasks experiencing larger losses receiving increased weights in the next iteration. Therefore, the joint loss function must be both inversely proportional to the rate of decrease and directly proportional to the current loss. Furthermore, considering the loss from self-supervised learning, two learnable parameters, ω and θ, are introduced to jointly incorporate the self-supervised learning process.

[0062] Understandably, in this embodiment, a dynamic weight adjustment strategy is adopted when constructing the joint learning framework involving six different main tasks and one auxiliary task. This strategy comprehensively considers the rate of loss decrease and the current loss value of each task. First, tasks with a slower rate of loss decrease are given more attention in the next iteration by increasing their weights. Simultaneously, tasks with larger current loss values ​​are equally important in training, and therefore their weights are increased accordingly. Furthermore, to further improve the model's generalization ability and learning performance, a self-supervised learning mechanism is introduced and combined with supervised learning tasks. Two learnable parameters are defined to adjust the contribution of the self-supervised learning loss to the joint loss function. The introduction of these two parameters provides the model with greater flexibility, enabling it to automatically adjust the weight ratio of self-supervised learning and supervised learning according to different training stages and data characteristics. This design not only improves the model's training efficiency but also helps to improve the model's final performance.

[0063] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that an article or device comprising a list of elements includes not only those elements but also other elements not expressly listed. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the article or device comprising said element. Terms such as "connected" or "linked" are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect. The orientations or positional relationships indicated by terms such as "upper," "lower," "left," and "right" are based on the orientations or positional relationships shown in the accompanying drawings and are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as limiting the invention.

[0064] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0065] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A remote sensing multi-task interpretation method of cross-knowledge interactive learning and self-supervised auxiliary learning, characterized in that, The method comprises the following steps: acquiring a to-be-processed image, setting a task requirement of the to-be-processed image, the task requirement comprising multiple types of single-modal data classification tasks, multi-modal data classification tasks, single-modal data segmentation tasks, multi-modal data segmentation tasks, single-modal data change detection tasks and multi-modal data change detection tasks; inputting the to-be-processed image into a trained multi-task interpretation network, and obtaining a result of the to-be-processed image after performing the task requirement according to the task requirement of the to-be-processed image; wherein the trained multi-task interpretation network is obtained by training an initial multi-task interpretation network in a self-supervised learning manner with preset category data as a training data set; before training the initial multi-task interpretation network, the method further comprises the following steps: constructing a multi-task interpretation network; wherein the multi-task interpretation network comprises an encoder, a classifier, a classification fusion module, a first neck network, a first feature fusion module, a second neck network, a second feature fusion module, a first shared decoder, a second shared decoder, a third shared decoder, a fourth shared decoder and a decoder; an input end of the encoder receives to-be-processed data, an output end of the encoder is connected with an input end of the classifier, the output end of the encoder is also connected with an input end of the classification fusion module, the output end of the encoder is also connected with an input end of the first neck network, the output end of the encoder is also connected with an input end of the first feature fusion module, the output end of the encoder is also connected with an input end of the second neck network, the output end of the encoder is also connected with an input end of the second feature fusion module, the output end of the encoder is also connected with an input end of the decoder, an output end of the decoder outputs reconstructed to-be-processed data, an output end of the first neck network is connected with an input end of the first shared decoder, an output end of the first shared decoder outputs a single-modal data segmentation result, an output end of the first feature fusion module is connected with an input end of the second shared decoder, an output end of the second shared decoder outputs a multi-modal data segmentation result, an output end of the second neck network is connected with an input end of the third shared decoder, an output end of the third shared decoder outputs a single-modal data change detection result, an output end of the second feature fusion module is connected with an input end of the fourth shared decoder, and an output end of the fourth shared decoder outputs a multi-modal data change detection result.

2. The remote sensing multi-task interpretation method of cross-knowledge interactive learning and self-supervised auxiliary learning according to claim 1, characterized in that, The classification fusion module comprises multiple classifiers; the first neck network and the second neck network comprise a feature pyramid network.

3. The method of claim 1, wherein, The trained multi-task interpretation network is obtained by training the initial multi-task interpretation network in a self-supervised learning manner with preset category data as a training data set, comprising the following steps: acquiring a training data set, the training data set comprising single-modal classification data sets, multi-modal classification data sets, single-modal segmentation data sets, multi-modal segmentation data sets, single-modal change detection data sets, multi-modal change detection data sets and task data sets; acquiring real labels corresponding to training samples in the training data set; The training data set and the real labels corresponding to the training samples in the training data set are input into an initial multi-task interpretation network for iterative training, and the value of a loss function reaches a minimum in a preset number of iterations, so as to obtain a trained multi-task interpretation network.

4. The remote sensing multi-task interpretation method of cross-knowledge interactive learning and self-supervised auxiliary learning according to claim 3, characterized in that, The training data set and the real labels corresponding to the training samples in the training data set are input into an initial multi-task interpretation network for iterative training, and the value of a loss function reaches a minimum in a preset number of iterations, so as to obtain a trained multi-task interpretation network. The single-modal classification data set is input into the encoder for processing to obtain first features, the first features are input into the classifier for normalization processing, the normalized first features are mapped to a category space to obtain a prediction score of each category of training samples in the single-modal classification data set, and a category probability distribution is generated through an activation function; by comparing the prediction score of each category of training samples in the single-modal classification data set with the real labels corresponding to the training samples in the single-modal classification data set, a loss of a single-modal data classification task is obtained. The single-modal segmentation data set is input into the encoder for processing to obtain multi-layer second features, the multi-layer second features are input into the first neck network for feature fusion, the fused features are input into the first shared decoder for multi-scale feature processing, and a single-modal data segmentation graph is obtained; by comparing the single-modal data segmentation graph with the real labels corresponding to the training samples in the single-modal segmentation data set, a loss of a single-modal data segmentation task is obtained. The single-modal change detection data set is input into the encoder for processing to obtain multi-layer third features, the multi-layer third features are input into the first feature fusion module for feature fusion to obtain fused features, the multi-layer third features are input into the decoder for processing to obtain decoding features, the fused features and the decoding features are subjected to absolute difference to obtain difference features between the fused features and the decoding features, and the difference features are input into the second shared decoder for processing to obtain a result of single-modal data change detection; by comparing the result of single-modal data change detection with the real labels corresponding to the training samples in the single-modal change detection data set, a loss of a single-modal data change detection task is obtained. The multi-modal classification data set is input into the encoder for processing to obtain a plurality of first features, the plurality of first features are input into different classifiers for processing to obtain a plurality of prediction scores of training samples of each category in the multi-modal classification data set, and the plurality of prediction scores are averaged; by comparing the average score of each category of training samples in the multi-modal classification data set with the real labels corresponding to the training samples in the multi-modal classification data set, a loss of a multi-modal data classification task is obtained. The multi-modal segmentation data set is input to the encoder for processing to obtain a plurality of layers of second features, the plurality of layers of second features are input to the second neck network for feature fusion to obtain a plurality of fused features, the plurality of fused features are input to the third shared decoder after averaging to obtain multi-scale feature processing, and a multi-modal data segmentation graph is obtained; by comparing the multi-modal data segmentation graph with the real label corresponding to the training sample in the multi-modal segmentation data set, the loss of the multi-modal data segmentation task is obtained; The multi-modal change detection data set is input to the encoder for processing to obtain a plurality of layers of third features, the plurality of layers of third features are input to the second feature fusion module for feature fusion to obtain a plurality of fused features, the plurality of layers of third features are input to the decoder for processing to obtain a plurality of decoding features, the fused features and the decoding features are subjected to absolute difference to obtain a plurality of difference features between the fused features and the decoding features, and the plurality of difference features are input to the fourth shared decoder after averaging for processing to obtain a multi-modal data change detection result; by comparing the multi-modal data change detection result with the real label corresponding to the training sample in the multi-modal change detection data set, the loss of the multi-modal data change detection task is obtained.

5. The method of claim 3, wherein, The training data set and the real label corresponding to the training sample in the training data set are input to an initial multi-task interpretation network for iterative training, including: The task data set is input to the encoder for mask processing, and the mask rate is 75%; the processed features are input to the decoder for feature extraction to obtain a reconstructed task data set; by comparing the features of the training sample in the task data set with the features of the training sample in the reconstructed task data set, a self-supervised loss is obtained.

6. The method of claim 4, wherein, The encoder processing process includes: The encoder divides the input data to be processed into a plurality of fixed-size image blocks and flattens each image block; the expanded image block is embedded with position information; The image blocks embedded with position information are processed by the encoding layers cascaded in the encoder to obtain long-distance dependency and context information in the image blocks, and a plurality of feature image blocks are obtained, which are shared in the single-modal data classification task, the single-modal data segmentation task, the single-modal data change detection task, the multi-modal data classification task, the multi-modal data segmentation task and the multi-modal data change detection task.

7. The method of claim 4, wherein, The first shared decoder and the second shared decoder share network parameters, and the third shared decoder and the fourth shared decoder share network parameters.

8. The method of claim 3, wherein, the loss function comprises a loss of a single modality data classification task a loss of a single modality data segmentation task a loss of a single modality data change detection task a loss of a multi-modality data classification task a loss of a multi-modality data segmentation task a loss of a multi-modality data change detection task and a self-supervised loss , the loss function is expressed as: ； wherein, , denotes the loss of the first iteration process, and denotes the learnable parameters, and both denote the loss of the iteration process.

9. A remote sensing multi-task interpretation device of cross-knowledge interactive learning and self-supervised auxiliary learning, characterized in that, It includes: An image acquisition module is configured to acquire an image to be processed and set task requirements for the image to be processed, the task requirements including a plurality of single-modal data classification tasks, multi-modal data classification tasks, single-modal data segmentation tasks, multi-modal data segmentation tasks, single-modal data change detection tasks and multi-modal data change detection tasks. An image processing module is configured to input the to-be-processed image into the trained multi-task interpretation network to obtain a result of the to-be-processed image after performing the task requirement according to the task requirement of the to-be-processed image. The trained multi-task interpretation network is trained in a self-supervised learning manner by taking preset category data as a training data set to obtain the trained multi-task interpretation network. Before training the initial multi-task interpretation network, the method further includes: constructing a multi-task interpretation network; The multi-task interpretation network includes an encoder, a classifier, a classification fusion module, a first neck network, a first feature fusion module, a second neck network, a second feature fusion module, a first shared decoder, a second shared decoder, a third shared decoder, a fourth shared decoder, and a decoder. The input end of the encoder receives to-be-processed data. The output end of the encoder is connected with the input end of the classifier. The output end of the encoder is also connected with the input end of the classification fusion module. The output end of the encoder is also connected with the input end of the first neck network. The output end of the encoder is also connected with the input end of the first feature fusion module. The output end of the encoder is also connected with the input end of the second neck network. The output end of the encoder is also connected with the input end of the second feature fusion module. The output end of the encoder is also connected with the input end of the decoder. The output end of the decoder outputs reconstructed to-be-processed data. The output end of the first neck network is connected with the input end of the first shared decoder. The output end of the first shared decoder outputs a single-modality data segmentation result. The output end of the first feature fusion module is connected with the input end of the second shared decoder. The output end of the second shared decoder outputs a multi-modality data segmentation result. The output end of the second neck network is connected with the input end of the third shared decoder. The output end of the third shared decoder outputs a single-modality data change detection result. The output end of the second feature fusion module is connected with the input end of the fourth shared decoder. The output end of the fourth shared decoder outputs a multi-modality data change detection result.

Images