Remote sensing multi-task interpretation method and device for cross-knowledge interactive learning and self-supervised auxiliary learning

Abstract

The invention discloses a remote sensing multi-task interpretation method and device for cross-knowledge interactive learning and self-supervised auxiliary learning, and relates to the technical field of remote sensing visual interpretation, and the method comprises the steps: obtaining a to-be-processed image, setting the task requirements of the to-be-processed image, and setting the task requirements of the to-be-processed image; the task requirements comprise one or more of a single-mode data classification task, a multi-mode data classification task, a single-mode data segmentation task, a multi-mode data segmentation task, a single-mode data change detection task and a multi-mode data change detection task; inputting a to-be-processed image into the trained multi-task interpretation network, and obtaining a result after the to-be-processed image executes the task requirement according to the task requirement of the to-be-processed image; wherein the trained multi-task interpretation network is obtained by training the initial multi-task interpretation network in a self-supervised learning mode by taking data of a preset category as a training data set. According to the method, remote sensing vision single-mode classification, segmentation and change detection and multi-mode classification, segmentation and change detection interpretation can be realized, and a high-precision interpretation result is obtained.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing visual interpretation, and specifically relates to a remote sensing multi-task interpretation method and device for cross-knowledge interactive learning and self-supervision assisted learning. Background Art

[0002] At present, multi-task learning, as an advanced machine learning paradigm, is showing extraordinary potential in many fields through its unique advantages. The core idea of ​​multi-task learning is to optimize multiple related tasks in parallel, utilize the complementarity between tasks, implement parameter sharing strategies in the shared network structure layer, and carefully construct a joint loss function to enhance the generalization ability of the model, ensure balanced performance improvement among multiple tasks, or achieve excellent performance on specific tasks with the help of auxiliary information from other tasks. Multi-task learning has made good progress in the field of natural images. However, in multi-task learning, different tasks often use different data sets, and the differences between different data sets will have a certain impact on model performance. In remote sensing images, due to the diversity of data set modalities, there are often significant differences between data sets, and these differences will have a greater impact on model performance. In order to realize a multi-task interpretation system for remote sensing, it is necessary to consider the differences in data sets and the balance of effects between different tasks. The existing technical system is still insufficient in dealing with these challenges in multi-task learning of remote sensing images.

[0003] In the prior art, Liu Shangwang et al. proposed a semantic segmentation of building objects in remote sensing images based on multi-task learning of Unet network, which can improve the accuracy of semantic segmentation of building objects in remote sensing images and has certain accuracy and timeliness; however, the depth of the proposed network is still limited, and the boundary distance uses a simple Euclidean distance. Ma Hui et al. proposed a multi-task learning deep twin network model, which is suitable for change detection scenarios with no changes, unchanged samples and historical thematic maps; however, the existing change detection accuracy does not meet the actual engineering application requirements and needs to be combined with manual interpretation.

[0004] Therefore, there is an urgent need to provide a remote sensing visual multi-task interpretation method to solve the above-mentioned defects in the prior art. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a remote sensing multi-task interpretation method and device for cross-knowledge interactive learning and self-supervision assisted learning. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] In a first aspect, the present invention provides a remote sensing multi-task interpretation method for cross-knowledge interactive learning and self-supervision assisted learning, comprising:

[0007] Acquire an image to be processed, and set task requirements for the image to be processed, where the task requirements include one or more of a single-modal data classification task, a multi-modal data classification task, a single-modal data segmentation task, a multi-modal data segmentation task, a single-modal data change detection task, and a multi-modal data change detection task;

[0008] Input the image to be processed into the trained multi-task interpretation network, and obtain the result after the image to be processed executes the task requirements according to the task requirements of the image to be processed;

[0009] The trained multi-task interpretation network uses preset category data as a training data set, and trains the initial multi-task interpretation network in a self-supervised learning manner.

[0010] In a second aspect, the present invention further provides a remote sensing multi-task interpretation device for cross-knowledge interactive learning and self-supervision assisted learning, comprising:

[0011] An image acquisition module is used to acquire an image to be processed and set task requirements for the image to be processed, wherein the task requirements include one or more of a single-modal data classification task, a multi-modal data classification task, a single-modal data segmentation task, a multi-modal data segmentation task, a single-modal data change detection task, and a multi-modal data change detection task;

[0012] The image processing module is used to input the image to be processed into the trained multi-task interpretation network, and obtain the result after the image to be processed executes the task requirements according to the task requirements of the image to be processed;

[0013] The trained multi-task interpretation network uses preset category data as a training data set, and trains the initial multi-task interpretation network in a self-supervised learning manner.

[0014] Beneficial effects of the present invention:

[0015] The present invention provides a remote sensing multi-task interpretation method and device for cross-knowledge interactive learning and self-supervised assisted learning. Aiming at the problems that the diversity of data set modalities affects the performance and the multi-task expansion is insufficient in the current remote sensing vision multi-task technology, the present invention aims to reduce the impact of the difference in data set modalities by merging data sets of multiple tasks and inputting them into a self-supervised learning network, and innovatively proposes partial encoder parameter sharing and a joint loss function of seven tasks to achieve cross-knowledge interactive learning between multiple tasks.

[0016] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1It is a flow chart of a remote sensing multi-task interpretation method of cross-knowledge interactive learning and self-supervision assisted learning provided by an embodiment of the present invention;

[0018] Figure 2 is a schematic diagram of a multi-task interpretation network provided by an embodiment of the present invention;

[0019] Figure 3 is a schematic diagram of a first neck network and a second neck network provided by an embodiment of the present invention;

[0020] Figure 4 It is a schematic diagram of an encoder provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0022] See also Figure 1 , Figure 1 It is a flow chart of a remote sensing multi-task interpretation method for cross-knowledge interactive learning and self-supervision assisted learning provided by an embodiment of the present invention. The remote sensing multi-task interpretation method for cross-knowledge interactive learning and self-supervision assisted learning provided by the present invention includes:

[0023] S101. Acquire an image to be processed and set task requirements for the image to be processed, where the task requirements include one or more of a unimodal data classification task, a multimodal data classification task, a unimodal data segmentation task, a multimodal data segmentation task, a unimodal data change detection task, and a multimodal data change detection task.

[0024] Specifically, in this embodiment, an image to be processed is obtained, and one or more of classification, segmentation or change detection is performed on the image to be processed according to task requirements.

[0025] S102, inputting the image to be processed into the trained multi-task interpretation network, and obtaining the result after the image to be processed executes the task requirements according to the task requirements of the image to be processed.

[0026] The trained multi-task interpretation network uses preset category data as a training data set, and trains the initial multi-task interpretation network in a self-supervised learning manner.

[0027] Specifically, see Figure 2 , Figure 2 is a schematic diagram of a multi-task interpretation network provided by an embodiment of the present invention. In this embodiment, before training the initial multi-task interpretation network, it also includes:

[0028] Build a multi-task interpretation network;

[0029] Among them, the multi-task interpretation network includes an encoder, a classifier, a classification fusion module, a first neck network, a first feature fusion module, a second neck network, a second feature fusion module, a first shared decoder, a second shared decoder, a third shared decoder, a fourth shared decoder and a decoder; the input end of the encoder receives the data to be processed, the output end of the encoder is connected to the input end of the classifier, the output end of the encoder is also connected to the input end of the classification fusion module, the output end of the encoder is also connected to the input end of the first neck network, the output end of the encoder is also connected to the input end of the first feature fusion module, the output end of the encoder is also connected to the input end of the second neck network, and the output end of the encoder is also connected to the input end of the second feature fusion module. The output end of the encoder is also connected to the input end of the decoder, the output end of the decoder outputs the reconstructed data to be processed, the output end of the first neck network is connected to the input end of the first shared decoder, the output end of the first shared decoder outputs the single-modal data segmentation result, the output end of the first feature fusion module is connected to the input end of the second shared decoder, the output end of the second shared decoder outputs the multi-modal data segmentation result, the output end of the second neck network is connected to the input end of the third shared decoder, the output end of the third shared decoder outputs the single-modal data change detection result, the output end of the second feature fusion module is connected to the input end of the fourth shared decoder, and the output end of the fourth shared decoder outputs the multi-modal data change detection result.

[0030] In this embodiment, see Figure 3 , Figure 3 It is a schematic diagram of a first neck network and a second neck network provided by an embodiment of the present invention. The classification fusion module includes multiple classifiers; the first neck network and the second neck network include feature pyramid networks.

[0031] It should be noted that the Feature Pyramid Network has been widely used in tasks such as object detection and semantic segmentation. Figure 3 As shown in the figure, the input image is subjected to feature extraction through the backbone network to obtain a series of feature maps of different scales; feature fusion operations are performed between different feature layers to obtain richer feature representations; and the fused feature maps are used for tasks such as target detection and semantic segmentation.

[0032] In this embodiment, please continue to refer to Figure 2 The trained multi-task interpretation network uses the preset category data as the training data set, and trains the initial multi-task interpretation network in a self-supervised learning manner, including:

[0033] Obtaining a training data set, the training data set includes a unimodal classification data set, a multimodal classification data set, a unimodal segmentation data set, a multimodal segmentation data set, a unimodal change detection data set, a multimodal change detection data set, and a task data set;

[0034] Get the true labels corresponding to the training samples in the training data set;

[0035] The training data set and the true labels corresponding to the training samples in the training data set are input into the initial multi-task interpretation network for iterative training. Within the preset number of iterations, the value of the loss function reaches the minimum, and a trained multi-task interpretation network is obtained.

[0036] It should be noted that, in this embodiment, the AdamW optimizer is used, the learning rate is set to 0.00006, and the weight decay is set to 0.01 for training, and the test is performed after the optimal epoch is obtained.

[0037] In this embodiment, the method further includes: organizing information of data sets of different tasks, and preprocessing the data sets to obtain data sets that meet the requirements.

[0038] Specifically, the information of the data set mainly includes the modality of the data set, the size of the image, the number of channels, etc. In order to share some encoder parameters in subsequent experiments, it is necessary to ensure that the size and number of channels of the data set input to the network are consistent.

[0039] In this example, all data are processed into three-channel data sets of size 512x512, and all data sets are scaled to a size of 512x512 through bilinear interpolation.

[0040] For the SAR mode, two common data are considered. One is a single-channel data set, which is converted into three-channel data by replication. The other is a two-channel data set consisting of vv and vh. The ratio of channel one to channel two is used as the third channel to form a three-channel data set.

[0041] For multispectral modality, the R, G, and B channels are selected to form a three-channel data set.

[0042] The processed datasets of the six main tasks are merged together as the dataset of the auxiliary task.

[0043] In this embodiment, see Figure 4 , Figure 4 is a schematic diagram of an encoder provided by an embodiment of the present invention, wherein a training data set and true labels corresponding to training samples in the training data set are input into an initial multi-task interpretation network for iterative training, including:

[0044] The unimodal classification dataset is input into the encoder for processing to obtain the first feature, the first feature is input into the classifier for normalization, the normalized first feature is mapped to the category space, the prediction score of the training sample of each category in the unimodal classification dataset is obtained, and the category probability distribution is generated through the activation function; the loss of the unimodal data classification task is obtained by comparing the prediction score of the training sample of each category in the unimodal classification dataset with the true label corresponding to the training sample in the unimodal classification dataset;

[0045] The unimodal segmentation data set is input into the encoder for processing to obtain multi-layer second features, the multi-layer second features are input into the first neck network for feature fusion, and the fused features are input into the first shared decoder for multi-scale feature processing to obtain a unimodal data segmentation map; the loss of the unimodal data segmentation task is obtained by comparing the unimodal data segmentation map with the true labels corresponding to the training samples in the unimodal segmentation data set; optionally, the first neck network is used to fuse the multi-layer features of the encoder, specifically fusing the 2nd, 5th, 8th, and 11th layers of features of ViT-Base, or fusing the 7th, 11th, 15th, and 23rd layers of features of ViT-Large; it can be understood that the obtained unimodal data segmentation map has the same size as the input image;

[0046] Input the unimodal change detection data set to the encoder for processing to obtain multi-layer third features, input the multi-layer third features to the first feature fusion module for feature fusion to obtain fused features, input the multi-layer third features to the decoder for processing to obtain decoded features, make an absolute difference between the fused features and the decoded features to obtain the difference features between the fused features and the decoded features, input the difference features to the second shared decoder for processing to obtain the results of unimodal data change detection; by comparing the results of unimodal data change detection with the true labels corresponding to the training samples in the unimodal change detection data set, the loss of the unimodal data change detection task is obtained; optionally, the first feature fusion network is used to fuse the multi-layer features of the encoder, specifically fusing the 2nd, 5th, 8th, and 11th layers of features of ViT-Base, or fusing the 7th, 11th, 15th, and 23rd layers of features of ViT-Large;

[0047] Input the multimodal classification dataset into the encoder for processing to obtain multiple first features, input the multiple first features into different classifiers for processing to obtain multiple prediction scores of the training samples of each category in the multimodal classification dataset, and average the multiple prediction scores; obtain the loss of the multimodal data classification task by comparing the average score of the training samples of each category in the multimodal classification dataset with the true labels corresponding to the training samples in the multimodal classification dataset;

[0048] The multimodal segmentation data set is input into the encoder for processing to obtain multi-layer second features, the multi-layer second features are input into the second neck network for feature fusion to obtain multiple fused features, the multiple fused features are averaged and input into the third shared decoder for multi-scale feature processing to obtain a multimodal data segmentation map; the loss of the multimodal data segmentation task is obtained by comparing the multimodal data segmentation map with the true labels corresponding to the training samples in the multimodal segmentation data set; optionally, the second neck network is used to fuse the multi-layer features of the encoder, specifically fusing the 2nd, 5th, 8th, and 11th layers of features of ViT-Base, or fusing the 7th, 11th, 15th, and 23rd layers of features of ViT-Large; it can be understood that the obtained multimodal data segmentation map has the same size as the input image;

[0049] The multimodal change detection data set is input into the encoder for processing to obtain multi-layer third features, the multi-layer third features are input into the second feature fusion module for feature fusion to obtain multiple fused features, the multi-layer third features are input into the decoder for processing to obtain multiple decoding features, the fused features and the decoding features are made into absolute differences to obtain multiple difference features between the fused features and the decoding features, the multiple difference features are averaged and input into the fourth shared decoder for processing to obtain the result of multimodal data change detection; the loss of the multimodal data change detection task is obtained by comparing the result of the multimodal data change detection with the true labels corresponding to the training samples in the multimodal change detection data set; optionally, the first feature fusion network is used to fuse the multi-layer features of the encoder, specifically fusing the 2nd, 5th, 8th, and 11th layers of features of ViT-Base, or fusing the 7th, 11th, 15th, and 23rd layers of features of ViT-Large, such as Figure 4 As shown, d=layers, counting starts from 0. It can be understood that four features are obtained by inputting into the decoder and the second neck network respectively, namely: the feature of modality 1 before change, the feature of modality 2 before change, the feature of modality 1 after change, and the feature of modality 2 after change; the following four features for inputting into the decoder are obtained: the absolute value difference feature of the feature of modality 2 after change and the feature of modality 1 before change, the absolute value difference feature of the feature of modality 2 after change and the feature of modality 2 before change, the absolute value difference feature of the feature of modality 1 after change and the feature of modality 1 before change, and the absolute value difference feature of the feature of modality 1 after change and the feature of modality 2 before change, and the average of these four features is obtained to obtain the difference feature.

[0050] In this embodiment, the training data set and the true labels corresponding to the training samples in the training data set are input into the initial multi-task interpretation network for iterative training, including:

[0051] The task dataset is input into the encoder for mask processing with a mask rate of 75%. The processed features are input into the decoder for feature extraction to obtain the reconstructed task dataset. The self-supervision loss is obtained by comparing the features of the training samples in the task dataset with the features of the training samples in the reconstructed task dataset.

[0052] In this embodiment, the encoder processing process implements the underlying shared parameter network, and the encoder processing process includes:

[0053] The encoder divides the input data to be processed into multiple fixed-size image blocks and flattens each image block; embeds the flattened image block into position information;

[0054] The image blocks with embedded position information are processed through the cascaded encoding layers in the encoder to obtain long-distance dependencies and contextual information in the image blocks, thereby obtaining multi-feature image blocks. The multi-feature image blocks are shared in single-modal data classification tasks, single-modal data segmentation tasks, single-modal data change detection tasks, multi-modal data classification tasks, multi-modal data segmentation tasks, and multi-modal data change detection tasks.

[0055] It is understandable that in this embodiment, in order to make full use of the potential correlation between multiple main task data sets and reduce the performance bottleneck caused by the differences in different modal data sets, an innovative strategy is proposed to comprehensively integrate the data sets of all main tasks to build a unified and diversified data set. Based on this integrated data set, an auxiliary self-supervised learning task is designed and implemented to explore the deep structure and inherent laws in the data in an unsupervised manner. In this framework, the sharing mechanism of encoder weights is adopted. By sharing encoder weights across tasks, it can not only promote the transfer and fusion of knowledge between different tasks, but also effectively reduce model parameters and improve training efficiency. More importantly, this weight sharing method enables the model to learn a more general and more generalized data representation that can cross the boundaries of different modalities and fields, and provide strong support for subsequent main task learning.

[0056] In this embodiment, the first shared decoder shares network parameters with the second shared decoder, and the third shared decoder shares network parameters with the fourth shared decoder to achieve high-level shared network parameters. For single-modal image segmentation and multi-modal image segmentation, the network parameters of the decoder UPerNet are shared; for single-modal change detection and multi-modal change detection, the network parameters of the decoder UPerNet are also shared.

[0057] It is understandable that in this embodiment, the same decoder UPerNet is used to process unimodal and multimodal tasks respectively. The total number of parameters of the model is significantly reduced, the training process is accelerated, and the model's demand for computing resources is reduced, which also reduces the computational cost during training. This is especially important for large data sets and complex networks. The decoder is shared between different tasks, and universal feature representations across tasks and modalities can be learned during the training process. This knowledge transfer helps improve the generalization ability of the model on unseen data or new tasks.

[0058] In this embodiment, the loss function includes the loss L1 of the unimodal data classification task, the loss L2 of the unimodal data segmentation task, the loss L3 of the unimodal data change detection task, the loss L4 of the multimodal data classification task, the loss L5 of the multimodal data segmentation task, the loss L6 of the multimodal data change detection task, and the self-supervision loss L m , the loss function L is expressed as:

[0059]

[0060] Among them, i∈[1,2,3,4,5,6], L i0 represents the loss of the first iteration process, ω and θ represent learnable parameters, L j and L i Both represent the loss of the iterative process.

[0061] It is understandable that, considering the decreasing rate of loss of different tasks, the slower decreasing rate will increase its weight in the next iteration. At the same time, considering the current loss of different tasks, the larger loss will increase its weight in the next iteration. Therefore, the joint loss function should be inversely proportional to the decreasing rate and proportional to the current loss. At the same time, considering the loss of self-supervised learning, two learnable parameters ω and θ are introduced to combine self-supervised learning.

[0062] It is understandable that in this embodiment, when constructing a joint learning framework involving six different main tasks and one auxiliary task, a dynamic weight adjustment strategy is adopted, which comprehensively considers the loss reduction rate and the current loss value of each task. First, for tasks with a slower loss reduction rate, more attention is given by increasing their weights in the next iteration. At the same time, tasks with larger current loss values ​​are equally important in training, so the weights of these tasks are also increased accordingly. In addition, in order to further improve the generalization ability and learning effect of the model, a self-supervised learning mechanism is introduced and combined with supervised learning tasks. Two learnable parameters are defined to adjust the contribution of self-supervised learning loss in the joint loss function. The introduction of these two parameters provides more flexibility for the model, enabling it to automatically adjust the weight ratio of self-supervised learning and supervised learning according to different training stages and data characteristics. This design not only improves the training efficiency of the model, but also helps to improve the final performance of the model.

[0063] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant is intended to cover non-exclusive inclusion, so that the article or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the article or device including the elements. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The orientation or position relationship indicated by "up", "down", "left", "right", etc. is based on the orientation or position relationship shown in the accompanying drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention.

[0064] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

[0065] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A remote sensing multi-task interpretation method based on cross-knowledge interactive learning and self-supervision assisted learning, characterized in that: include: Acquire an image to be processed, and set a task requirement for the image to be processed, wherein the task requirement includes one or more of a unimodal data classification task, a multimodal data classification task, a unimodal data segmentation task, a multimodal data segmentation task, a unimodal data change detection task, and a multimodal data change detection task; Inputting the image to be processed into a trained multi-task interpretation network, and obtaining the result of the image to be processed after executing the task requirements according to the task requirements of the image to be processed; The trained multi-task interpretation network is obtained by training the initial multi-task interpretation network in a self-supervised learning manner using data of preset categories as a training data set.

2. The remote sensing multi-task interpretation method of cross-knowledge interactive learning and self-supervision assisted learning according to claim 1 is characterized in that: Before training the initial multi-task interpretation network, it also includes: Build a multi-task interpretation network; Wherein, the multi-task interpretation network includes an encoder, a classifier, a classification fusion module, a first neck network, a first feature fusion module, a second neck network, a second feature fusion module, a first shared decoder, a second shared decoder, a third shared decoder, a fourth shared decoder and a decoder; the input end of the encoder receives the data to be processed, the output end of the encoder is connected to the input end of the classifier, the output end of the encoder is also connected to the input end of the classification fusion module, the output end of the encoder is also connected to the input end of the first neck network, the output end of the encoder is also connected to the input end of the first feature fusion module, the output end of the encoder is also connected to the input end of the second neck network, and the output end of the encoder is also connected to the input end of the second feature fusion module , the output end of the encoder is also connected to the input end of the decoder, and the output end of the decoder outputs the reconstructed data to be processed, the output end of the first neck network is connected to the input end of the first shared decoder, and the output end of the first shared decoder outputs a single-modal data segmentation result, the output end of the first feature fusion module is connected to the input end of the second shared decoder, and the output end of the second shared decoder outputs a multi-modal data segmentation result, the output end of the second neck network is connected to the input end of the third shared decoder, and the output end of the third shared decoder outputs a single-modal data change detection result, the output end of the second feature fusion module is connected to the input end of the fourth shared decoder, and the output end of the fourth shared decoder outputs a multi-modal data change detection result.

3. The remote sensing multi-task interpretation method of cross-knowledge interactive learning and self-supervision assisted learning according to claim 2 is characterized in that: The classification fusion module includes multiple classifiers; the first neck network and the second neck network include feature pyramid networks.

4. The remote sensing multi-task interpretation method of cross-knowledge interactive learning and self-supervision assisted learning according to claim 2 is characterized in that: The trained multi-task interpretation network uses the preset category data as the training data set, and trains the initial multi-task interpretation network in a self-supervised learning manner, including: Acquire a training data set, wherein the training data set includes a unimodal classification data set, a multimodal classification data set, a unimodal segmentation data set, a multimodal segmentation data set, a unimodal change detection data set, a multimodal change detection data set, and a task data set; Get the true labels corresponding to the training samples in the training data set; The training data set and the true labels corresponding to the training samples in the training data set are input into the initial multi-task interpretation network for iterative training. Within a preset number of iterations, the value of the loss function reaches a minimum, and a trained multi-task interpretation network is obtained.

5. The remote sensing multi-task interpretation method of cross-knowledge interactive learning and self-supervision assisted learning according to claim 4 is characterized in that: Inputting the training data set and the true labels corresponding to the training samples in the training data set into the initial multi-task interpretation network for iterative training, including: Input the unimodal classification dataset to the encoder for processing to obtain a first feature, input the first feature to the classifier for normalization, map the normalized first feature to the category space, obtain the prediction score of the training sample of each category in the unimodal classification dataset, and generate the category probability distribution through the activation function; obtain the loss of the unimodal data classification task by comparing the prediction score of the training sample of each category in the unimodal classification dataset with the true label corresponding to the training sample in the unimodal classification dataset; The unimodal segmentation data set is input into the encoder for processing to obtain multi-layer second features, the multi-layer second features are input into the first neck network for feature fusion, and the fused features are input into the first shared decoder for multi-scale feature processing to obtain a unimodal data segmentation map; the loss of the unimodal data segmentation task is obtained by comparing the unimodal data segmentation map with the true labels corresponding to the training samples in the unimodal segmentation data set; Input the unimodal change detection data set to the encoder for processing to obtain multiple layers of third features, input the multiple layers of the third features to the first feature fusion module for feature fusion to obtain fused features, input the multiple layers of the third features to the decoder for processing to obtain decoded features, make an absolute difference between the fused features and the decoded features to obtain a difference feature between the fused features and the decoded features, input the difference feature to the second shared decoder for processing to obtain a result of unimodal data change detection; obtain the loss of the unimodal data change detection task by comparing the result of the unimodal data change detection with the true labels corresponding to the training samples in the unimodal change detection data set; Inputting the multimodal classification dataset into the encoder for processing to obtain a plurality of first features, inputting the plurality of first features into different classifiers for processing to obtain a plurality of prediction scores of the training samples of each category in the multimodal classification dataset, and averaging the plurality of prediction scores; obtaining the loss of the multimodal data classification task by comparing the average score of the training samples of each category in the multimodal classification dataset with the true labels corresponding to the training samples in the multimodal classification dataset; Input the multimodal segmentation data set to the encoder for processing to obtain multiple layers of second features, input the multiple layers of the second features to the second neck network for feature fusion to obtain multiple fused features, take the average of the multiple fused features and input them to the third shared decoder for multi-scale feature processing to obtain a multimodal data segmentation map; compare the multimodal data segmentation map with the true labels corresponding to the training samples in the multimodal segmentation data set to obtain the loss of the multimodal data segmentation task; The multimodal change detection data set is input into the encoder for processing to obtain multiple layers of third features, the multiple layers of the third features are input into the second feature fusion module for feature fusion to obtain multiple fused features, the multiple layers of the third features are input into the decoder for processing to obtain multiple decoded features, the fused features and the decoded features are made an absolute difference to obtain multiple difference features between the fused features and the decoded features, the multiple difference features are averaged and input into the fourth shared decoder for processing to obtain the result of multimodal data change detection; the loss of the multimodal data change detection task is obtained by comparing the result of the multimodal data change detection with the true labels corresponding to the training samples in the multimodal change detection data set.

6. The remote sensing multi-task interpretation method of cross-knowledge interactive learning and self-supervision assisted learning according to claim 4 is characterized in that: Inputting the training data set and the true labels corresponding to the training samples in the training data set into the initial multi-task interpretation network for iterative training, including: The task data set is input into the encoder for mask processing with a mask rate of 75%, and the processed features are input into the decoder for feature extraction to obtain a reconstructed task data set; and the self-supervision loss is obtained by comparing the features of the training samples in the task data set with the features of the training samples in the reconstructed task data set.

7. The remote sensing multi-task interpretation method of cross-knowledge interactive learning and self-supervision assisted learning according to claim 5 is characterized in that: The encoder processing process includes: The encoder divides the input data to be processed into a plurality of image blocks of fixed size, and flattens each image block; embeds the flattened image block into the position information; The image blocks with embedded position information are processed by the cascaded encoding layers in the encoder to obtain long-distance dependencies and context information in the image blocks, so as to obtain multi-feature image blocks. The multi-feature image blocks are shared in the single-modal data classification task, the single-modal data segmentation task, the single-modal data change detection task, the multi-modal data classification task, the multi-modal data segmentation task and the multi-modal data change detection task.

8. The remote sensing multi-task interpretation method of cross-knowledge interactive learning and self-supervision assisted learning according to claim 5 is characterized in that: The first shared decoder shares network parameters with the second shared decoder, and the third shared decoder shares network parameters with the fourth shared decoder.

9. The remote sensing multi-task interpretation method of cross-knowledge interactive learning and self-supervision assisted learning according to claim 4 is characterized in that: The loss function includes the loss L1 of the unimodal data classification task, the loss L2 of the unimodal data segmentation task, the loss L3 of the unimodal data change detection task, the loss L4 of the multimodal data classification task, the loss L5 of the multimodal data segmentation task, the loss L6 of the multimodal data change detection task and the self-supervision loss L m , the loss function L is expressed as: Among them, i∈[1,2,3,4,5,6], L i0 represents the loss of the first iteration process, ω and θ represent learnable parameters, L j and L i Both represent the loss of the iterative process.

10. A remote sensing multi-task interpretation device with cross-knowledge interactive learning and self-supervision assisted learning, characterized in that: include: An image acquisition module is used to acquire an image to be processed and set a task requirement for the image to be processed, wherein the task requirement includes one or more of a unimodal data classification task, a multimodal data classification task, a unimodal data segmentation task, a multimodal data segmentation task, a unimodal data change detection task, and a multimodal data change detection task; An image processing module is used to input the image to be processed into a trained multi-task interpretation network, and obtain the result of the image to be processed after executing the task requirements according to the task requirements of the image to be processed; The trained multi-task interpretation network is obtained by training the initial multi-task interpretation network in a self-supervised learning manner using data of preset categories as a training data set.

Images