A remote sensing multimodal image segmentation method based on recurrent dual-task reciprocal learning

Abstract

The present invention relates to a remote sensing multimodal image segmentation method based on cyclic dual-task reciprocal learning. The present invention includes the following steps: constructing a multimodal remote sensing image segmentation task dataset and a synthesis task dataset; constructing a segmentation network based on hyperspectral and lidar data fusion; constructing a hyperspectral synthesis network based on a land object classification map and a multispectral image; constructing a cyclic dual-task reciprocal learning framework including segmentation and synthesis; training a multimodal image segmentation network based on cyclic dual-task reciprocal learning and obtaining results. Compared with the prior art, the present invention uses a cyclic dual-task reciprocal learning framework to promote each other between the segmentation network and the synthesis network, effectively improving the detail retention of the segmentation results and the consistency of cross-modal features, and enhancing the adaptability of the network in the case of modality loss, and can provide a stable segmentation effect in the case of incomplete information and modality misalignment, and is suitable for multi-source remote sensing data fusion processing and image segmentation tasks.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal remote sensing image processing, and in particular to a remote sensing multimodal image segmentation method based on cyclic dual-task reciprocal learning. Background Art

[0002] In recent years, with the continuous advancement of remote sensing technology, multimodal remote sensing images have been widely used in environmental monitoring, urban planning, agricultural management and other fields. Multimodal remote sensing images combine information obtained by different sensors, such as hyperspectral images and lidar data, and can provide richer ground features and spatial information, which provides strong support for the analysis and application of remote sensing data.

[0003] However, although multimodal remote sensing images have obvious advantages in information acquisition, their processing and analysis still face many challenges. Among them, the segmentation task of remote sensing images is particularly critical, because the segmentation results directly affect the subsequent object recognition and classification accuracy. Traditional remote sensing image segmentation methods usually rely on single-modal data and find it difficult to fully utilize the complementarity of multimodal information, resulting in poor segmentation effects in complex scenes. In addition, alignment issues between modalities and modality loss also bring additional challenges to the segmentation task.

[0004] In order to solve these problems, researchers have gradually explored multimodal image segmentation methods based on deep learning. However, existing multimodal learning frameworks often lack effective inter-task collaboration mechanisms, resulting in insufficient knowledge transfer between segmentation models and synthesis models, affecting the detail retention of segmentation results and the consistency of cross-modal features. Therefore, a new method is urgently needed to promote mutual enhancement between segmentation and synthesis tasks through an effective task reciprocal learning framework to improve the stability and accuracy of segmentation results.

[0005] Based on this, a remote sensing multimodal image segmentation method based on cyclic dual-task reciprocal learning is proposed. This method realizes the cyclic learning of segmentation and synthesis tasks by constructing a segmentation network that fuses hyperspectral and lidar data, and a hyperspectral synthesis network based on object classification maps and multispectral images. This innovative method not only improves the adaptability of the segmentation network in the case of modality loss, but also enhances its stability in the case of modality loss and modality misalignment, promoting the further development of multi-source remote sensing data fusion processing and image segmentation tasks. Summary of the invention

[0006] The purpose of the present invention is to overcome the limitations of existing methods, improve the detail retention of segmentation results and the consistency of cross-modal features and the adaptability of the network in the absence of modality, and provide a remote sensing multimodal image segmentation method based on cyclic dual-task reciprocal learning.

[0007] In order to achieve the above object, the technical solution of the present invention is as follows:

[0008] A remote sensing multimodal image segmentation method based on cyclic dual-task reciprocal learning includes the following steps:

[0009] 11) Construct multimodal remote sensing image segmentation task dataset and synthetic task dataset: obtain paired hyperspectral images and LiDAR images Used to construct segmentation task datasets; obtain paired object classification maps , Multispectral Imagery Used to construct synthetic task datasets; further perform data augmentation on segmentation task datasets and synthetic task datasets to expand the datasets;

[0010] 12) Construct a segmentation network based on hyperspectral and lidar data fusion: Construct an augmented hyperspectral image with paired input , augmented LiDAR images The segmentation network of the algorithm consists of an encoder and a decoder;

[0011] 13) Construct a hyperspectral synthesis network based on object classification map and multispectral image: Construct a hyperspectral synthesis network with augmented object classification map as input , Augmented Multispectral Imagery A hyperspectral synthesis network, in which the model adopts a conditional diffusion generation model;

[0012] 14) Construct a reciprocal dual-task learning framework including segmentation and synthesis: Construct a reciprocal dual-task learning framework for image segmentation and hyperspectral synthesis, in which the segmentation results are used as the input of the synthesis network and the synthesis results are used as the input of the segmentation network;

[0013] 15) Training a multimodal image segmentation network based on cyclic dual-task reciprocal learning and obtaining results: Train the remote sensing multimodal image segmentation network through a training paradigm that combines supervised learning and semi-supervised learning, and obtain the final test results.

[0014] The construction of a segmentation network based on hyperspectral and lidar data fusion includes the following steps:

[0015] 21) In the segmentation task dataset, there are corresponding ground object classification map labels The data is then used to generate hyperspectral images , LiDAR images and feature classification map labels Splice together by channel dimension;

[0016] 211) According to the size of 32×32, the spliced ​​hyperspectral image , LiDAR images and feature classification map labels Divide into multiple image blocks of corresponding sizes;

[0017] 212) For each segmented image block , randomly select an image block from the remaining multiple image blocks ,Will A sub-image block of a random size selected from the image block is used to replace the image block The sub-image block at the corresponding position in ;

[0018] 213) All image blocks processed by 1112) are divided into corresponding augmented hyperspectral images according to the channel dimension , augmented LiDAR images and augmented feature classification map labels , as a segmentation task dataset after data augmentation;

[0019] 22) In the synthetic task dataset, for the corresponding hyperspectral image labels The data is used to classify the features into , Multispectral Imagery and hyperspectral imagery tags Splice together by channel dimension;

[0020] 221) According to the size of 32×32, the spliced ​​land feature classification map , Multispectral Imagery and hyperspectral imagery tags Divide into multiple image blocks of corresponding sizes;

[0021] 222) For each image block after segmentation , randomly select an image block from the remaining multiple image blocks ,Will A sub-image block of random size is selected from The sub-image block at the corresponding position in ;

[0022] 223) All image blocks processed by 1122) are divided into corresponding augmented object classification maps according to the channel dimension , Augmented Multispectral Imagery and augmented hyperspectral imagery labels , as a synthetic task dataset after data augmentation.

[0023] The construction of a segmentation network based on hyperspectral and lidar data fusion includes the following steps:

[0024] 31) A segmentation network based on hyperspectral and lidar data fusion includes an encoder, a decoder and a bottleneck layer, and the encoder and the decoder are connected through a Fourier transform-based structure to maintain an attention jump connection structure;

[0025] 311) The encoder of the segmentation network is set to include 4 consecutive downsampling structures: the downsampling structure is a convolution layer with a convolution kernel of 3×3, a batch normalization layer, a Relu layer, a convolution layer with a convolution kernel of 3×3, a batch normalization layer, a Relu layer stack, and finally a 2x maximum pooling layer; paired augmented hyperspectral image , augmented LiDAR images The input is spliced ​​together according to the channel dimension, and four outputs of different sizes are obtained through four consecutive downsampling structures. , , , ;

[0026] 312) Set the bottleneck layer of the segmentation network: the bottleneck layer receives the output of the fourth downsampling structure As input, the structure is a fully connected layer, a Gelu layer, a fully connected layer, and a Gelu layer; the output of the bottleneck layer is ;

[0027] 313) The decoder of the segmentation network is assumed to consist of 4 consecutive upsampling structures: the first upsampling structure receives the output of the bottleneck layer As input, the upsampling structure is a convolution layer with a convolution kernel of 3×3, a batch normalization layer, a Relu layer, a convolution layer with a convolution kernel of 3×3, a batch normalization layer, and a Relu layer stack, and finally a transposed convolution with a stride of 2 to complete the upsampling operation; after four consecutive upsampling structures, four outputs of different sizes are obtained respectively , , , ;

[0028] 314) Set the Fourier transform-based structure-preserving attention jump connection structure connecting the encoder and decoder: the four downsampling structures of the encoder obtain outputs of four different sizes respectively, and the output of each size is connected through the structure-preserving attention jump connection structure; assuming that the feature map is , then the Fourier transform process can be expressed by the following formula:

[0029] ,

[0030] in represents Fourier transform, represents the phase, represents the amplitude, and Represents the horizontal and vertical frequency coordinates in the frequency domain;

[0031] The phase represents the structural information of the feature map, which is a tensor of the same size that can be used for gradient descent. Perform element-by-element multiplication and then perform inverse Fourier transform to convert it into a feature map , the process can be expressed by the following formula:

[0032] ,

[0033] in stands for the inverse Fourier transform, Represents element-wise multiplication:

[0034] Output of four downsampling structures , , , After the structure-preserving attention jump connection structure, four feature maps are obtained respectively. , , , ,

[0035] Spliced ​​to the first output of the bottleneck layer ,

[0036] Spliced ​​to the first output of the decoder ,

[0037] Spliced ​​to the second output of the decoder ,

[0038] Spliced ​​to the third output of the decoder ;

[0039] The feature map obtained by the fourth upsampling structure Input to the tanh activation function;

[0040] 32) The hyperspectral data and lidar data are spliced ​​in the channel dimension and input into the segmentation network based on the fusion of hyperspectral and lidar data to obtain the final prediction map of the land object classification results.

[0041] The construction of a hyperspectral synthesis network based on a ground object classification map and multispectral images comprises the following steps:

[0042] 41) A hyperspectral synthesis network based on ground object classification maps and multispectral images uses a conditional diffusion generation model as a synthesis framework:

[0043] 411) Network input is the augmented feature classification map and augmented multispectral imagery , and its corresponding augmented hyperspectral image label is ;

[0044] 412) Construct a standard Unet network to predict the mean of the multidimensional Gaussian distribution in the denoising process;

[0045] 413) In the forward process of diffusion from hyperspectral image to noise, the maximum number of steps for adding noise is ,but Represents the hyperspectral image The feature map obtained by forward diffusion of steps, where is less than A positive integer from Sampling from a normal distribution The forward diffusion process representing one step can be expressed by the following formula;

[0046] ,

[0047] ,

[0048] in Uniformly distributed between 0.0001 and 0.02, representing the intensity of adding noise at each step; represents a state with a mean of , the variance is Normal distribution of represents the identity matrix; represents a standard normal distribution;

[0049] 414) Randomly select steps ,from Generated by The vector obtained by adding noise , which can be expressed by the following formula:

[0050] ,

[0051] in ,and It follows a uniform distribution between 0.0001 and 0.02. ; Obey standard normal distribution;

[0052] 415) from Recover from The ideal mean It can be expressed by the following formula:

[0053] ,

[0054] in It obeys the standard normal distribution;

[0055] from Recover from The ideal variance It can be expressed by the following formula:

[0056] ,

[0057] In the reverse diffusion process of generating hyperspectral images, the denoising process can be expressed by the following formula:

[0058] ,

[0059] The reverse diffusion mean ,variance The value of is determined by the vector , Diffusion step index and augmented feature classification map , Augmented Multispectral Imagery Predict through Unet network;

[0060] 416) Execution The denoising operation generates the result , the denoising process of each step can be expressed by the following formula:

[0061] ,

[0062] in The value is less than greater than 0, Represents the Unet network.

[0063] The construction of a recurrent dual-task reciprocal learning framework including segmentation and synthesis includes the following steps:

[0064] 51) A recurrent dual-task reciprocal learning framework is established by constructing a segmentation network based on hyperspectral and lidar data fusion and a hyperspectral synthesis network based on object classification maps and multispectral images. The results of the segmentation network are used as the input of the synthesis network, and the results of the synthesis network are also used as the input of the segmentation network.

[0065] 52) For the segmentation network, the network input is the augmented hyperspectral image and augmented LiDAR images , the network output is a land feature classification map ; Its segmentation loss function is designed as follows:

[0066] 521) For paired augmented hyperspectral images and augmented LiDAR images There is a corresponding augmented feature classification map label data, through the cross entropy loss in supervised learning The training segmentation network parameters can be expressed by the following formula:

[0067] ,

[0068] in Represents the cross entropy loss calculation process;

[0069] 53) For the synthetic network, the network input is the augmented object classification map , Augmented Multispectral Imagery , the network output is a hyperspectral image ; Its synthetic loss function is designed as follows:

[0070] 531) For paired augmented feature classification maps , Augmented Multispectral Imagery Has its corresponding augmented hyperspectral image label data, through the mean squared error loss in supervised learning The training segmentation network parameters can be expressed by the following formula:

[0071] ,

[0072] in Represents the mean square error loss calculation process;

[0073] 54) One cycle is defined as the segmentation network and the synthesis network performing a forward propagation in sequence, where the result of the segmentation network is used as the input of the synthesis network, and the result of the synthesis network is also used as the input of the segmentation network; for data with missing modalities, that is, the augmented hyperspectral image is missing in the segmentation network , the synthetic network lacks augmented object classification maps The consistency loss is calculated by two cycles:

[0074] 541) For the lack of augmented hyperspectral images in segmentation networks , and there are paired augmented lidar images and augmented feature classification map labels The data is first forward propagated through the synthetic network to generate a synthetic hyperspectral image. , and then generate through a forward propagation of the segmentation network ; Repeat the above cycle again to generate , calculate its consistency loss , which can be expressed by the following formula:

[0075] ,

[0076] in Represents the mean square error loss calculation process;

[0077] 542) For the lack of augmented object classification maps in the synthetic network , and there are paired augmented multispectral images and augmented hyperspectral imagery labels The data is first segmented and forward propagated to generate a classification map of the ground objects. , and then generate through a forward propagation of the synthetic network ; Repeat the above cycle again to generate , calculate its consistency loss , which can be expressed by the following formula:

[0078] ,

[0079] in Represents the mean square error loss calculation process.

[0080] The training of a multimodal image segmentation network based on cyclic dual-task reciprocal learning and obtaining results include the following steps:

[0081] 61) Pass , , and Perform back propagation, where , is used to update the model parameters of the segmentation network, , Used to update the model parameters of the synthesis network;

[0082] 62) Determine whether the set number of training rounds has been reached. If so, complete the training of the multimodal image segmentation network based on cyclic dual-task reciprocal learning. Otherwise, continue training.

[0083] 63) The hyperspectral image and lidar image are spliced ​​according to the channel dimension, and the final segmentation result image is obtained through the segmentation network.

[0084] Beneficial Effects

[0085] The remote sensing multimodal image segmentation method based on cyclic dual-task reciprocal learning proposed in the present invention has significant innovation and superiority compared to the existing technology. First, a cyclic dual-task reciprocal learning framework of the segmentation network and the synthesis network is constructed, so that the segmentation and synthesis tasks promote each other, thereby improving the detail retention of the segmentation results and the consistency of cross-modal features. This method can effectively deal with complex situations such as incomplete information and modality misalignment, improve the adaptability and stability of the network when the modality is missing, and ensure the high accuracy and stability of the segmentation results. Through the design of this framework, when different modal data are missing, other available modal data can be fully utilized to supplement, thereby effectively improving the segmentation effect.

[0086] In addition, the present invention gives full play to the advantages of hyperspectral and lidar data in multimodal remote sensing data fusion, and enhances the model's ability to process complex remote sensing image data through refined network architecture design. In particular, by introducing a Fourier transform-based structure to maintain attention jump connections, the information flow of feature maps is optimized, and the performance of the segmentation network in complex image data is improved. At the same time, the present invention combines the training paradigms of supervised learning and semi-supervised learning, reduces dependence on large amounts of labeled data, and reduces manual labeling costs. This method is more efficient and stable in remote sensing image segmentation, and has strong generalization capabilities. It can be widely used in agriculture, environmental monitoring, geological exploration and other fields, and has significant application prospects and practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 It is a remote sensing multimodal image segmentation method based on cyclic dual-task reciprocal learning;

[0088] Figure 2 It is a segmentation network framework diagram based on hyperspectral and laser radar data fusion involved in the present invention;

[0089] Figure 3 The invention relates to a hyperspectral synthesis network framework diagram based on a land feature classification map and multispectral images;

[0090] Figure 4 The invention relates to a reciprocal learning framework diagram of a cyclic dual-task including segmentation and synthesis; DETAILED DESCRIPTION

[0091] In order to have a further understanding and recognition of the structural features and the effects achieved by the present invention, a preferred embodiment and accompanying drawings are used for detailed description as follows:

[0092] like Figure 1 As shown, the remote sensing multimodal image segmentation method based on cyclic dual-task reciprocal learning described in the present invention is characterized by comprising the following steps:

[0093] The first step is to build a multimodal dataset for remote sensing image segmentation:

[0094] Acquire pairs of hyperspectral images and LiDAR images Used for segmentation task dataset; obtain paired object classification maps , Multispectral Imagery Used for synthetic task datasets; perform data enhancement on the segmentation task dataset and the synthetic task dataset to expand the data. The specific steps are as follows:

[0095] (1) Constructing a multimodal dataset for remote sensing image segmentation: Obtaining paired hyperspectral images and LiDAR images Used for segmentation task dataset; obtain paired object classification maps , Multispectral Imagery Used for synthetic task datasets; perform data augmentation on segmentation task datasets and synthetic task datasets to expand the datasets:

[0096] (1-1) In the segmentation task dataset, there are corresponding ground object classification map labels The data is then used to generate hyperspectral images , LiDAR images and feature classification map labels Splice together by channel dimension;

[0097] (1-1-1) According to the size of 32×32, stitch the hyperspectral image , LiDAR images and feature classification map labels Divide into image blocks of corresponding size;

[0098] (1-1-2) For each image block after segmentation , randomly select an image block from the remaining image blocks ,Will A randomly selected sub-image block of random size is replaced The sub-image blocks at the corresponding geometric positions in ;

[0099] (1-1-3) All the image blocks after segmentation and replacement are segmented into corresponding hyperspectral images, lidar images and ground object classification map labels according to the channel dimension as the segmentation task dataset after data enhancement;

[0100] (1-2) In the synthetic task dataset, there are corresponding hyperspectral image labels The data is used to classify the features into , Multispectral Imagery and hyperspectral imagery tags Splice together by channel dimension;

[0101] (1-2-1) According to the size of 32×32, the spliced ​​feature classification map , Multispectral Imagery and hyperspectral imagery tags Divide into image blocks of corresponding size;

[0102] (1-2-2) For each image block after segmentation , randomly select an image block from the remaining image blocks ,Will A randomly selected sub-image block of random size is replaced The sub-image blocks at the corresponding geometric positions in ;

[0103] (1-2-3) All processed image blocks are divided into corresponding ground object classification maps, multispectral images and hyperspectral image labels according to the channel dimension as the synthetic task dataset after data enhancement;

[0104] The second step is to build a segmentation network based on the fusion of hyperspectral and lidar data:

[0105] The specific steps are as follows:

[0106] (1) The segmentation network based on hyperspectral and lidar data fusion consists of an encoder, a decoder, and a bottleneck layer, and the encoder and decoder are connected with attention skip connections through a Fourier transform-based structure;

[0107] (1-1) The encoder of the segmentation network is set to include 4 consecutive downsampling structures: the downsampling structure is a convolution layer with a convolution kernel of 3×3, a batch normalization layer, a Relu layer, a convolution layer with a convolution kernel of 3×3, a batch normalization layer, a Relu layer stack, and finally a 2x maximum pooling layer; paired hyperspectral images , LiDAR images The input is spliced ​​together according to the channel dimension, and four outputs of different sizes are obtained through four consecutive downsampling structures. , , , ;

[0108] (1-2) Set the bottleneck layer of the segmentation network: The bottleneck layer receives the output of the fourth downsampling structure As input, the structure is a fully connected layer, a Gelu layer, a fully connected layer, and a Gelu layer; the output of the bottleneck layer is ;

[0109] (1-3) The decoder of the segmentation network is set to include 4 consecutive upsampling structures: the first upsampling structure receives the output of the bottleneck layer As input, the upsampling structure is a convolution layer with a convolution kernel of 3×3, a batch normalization layer, a Relu layer, a convolution layer with a convolution kernel of 3×3, a batch normalization layer, and a Relu layer stack, and finally a transposed convolution with a stride of 2 to complete the upsampling operation; after four consecutive upsampling structures, four outputs of different sizes are obtained respectively , , , ;

[0110] (1-4) Set the Fourier transform-based structure-preserving attention jump connection structure connecting the encoder and decoder: the four downsampling structures of the encoder obtain outputs of four different sizes respectively, and the output of each size is subjected to the structure-preserving attention mechanism; assuming that the feature map is , then the Fourier transform process can be expressed by the following formula:

[0111] ,

[0112] in represents Fourier transform, represents the phase, represents the amplitude, and Represents the horizontal and vertical frequency coordinates in the frequency domain;

[0113] The phase represents the structural information of the feature map, which is a tensor of the same size that can be used for gradient descent. Perform element-by-element multiplication and then perform inverse Fourier transform to convert it into a feature map , the process can be expressed by the following formula:

[0114] ,

[0115] in stands for the inverse Fourier transform, Represents element-wise multiplication:

[0116] Output of four downsampling structures , , , After the structure-preserving attention mechanism, four feature maps are obtained. , , , ,

[0117] Spliced ​​to the first output of the bottleneck layer ,

[0118] Spliced ​​to the first output of the decoder ,

[0119] Spliced ​​to the second output of the decoder ,

[0120] Spliced ​​to the third output of the decoder ;

[0121] The feature map obtained by the fourth upsampling structure Input to the tanh activation function;

[0122] (2) The hyperspectral data and lidar data are spliced ​​in the channel dimension and input into the segmentation network based on the fusion of hyperspectral and lidar data to obtain the final prediction map of the land object classification results.

[0123] The third step is to build a hyperspectral synthesis network based on the object classification map and multispectral imagery:

[0124] The specific steps are as follows:

[0125] (1) A hyperspectral synthesis network based on ground object classification maps and multispectral images uses a conditional diffusion generation model as the synthesis framework:

[0126] (1-1) Network input is a land feature classification map and multispectral imaging , and its corresponding label hyperspectral image is ;

[0127] (1-2) Construct a standard Unet network to predict the mean of the multidimensional Gaussian distribution in the denoising process;

[0128] (1-3) In the forward process of diffusion from hyperspectral image to noise, the maximum number of steps of adding noise is ,but Represents the hyperspectral image The feature map obtained by forward diffusion of steps, where is less than A positive integer from Sampling from a normal distribution The forward diffusion process representing one step can be expressed by the following formula;

[0129] ,

[0130] ,

[0131] in Uniformly distributed between 0.0001 and 0.02, representing the intensity of adding noise at each step; represents a state with a mean of , the variance is Normal distribution of represents the identity matrix; represents a standard normal distribution;

[0132] (1-4) Randomly select the number of steps ,from Generated by The vector obtained by adding noise , which can be expressed by the following formula:

[0133] ,

[0134] in ,and It follows a uniform distribution between 0.0001 and 0.02. ; Obey standard normal distribution;

[0135] (1-5) From Recover from The ideal mean It can be expressed by the following formula:

[0136] ,

[0137] in It obeys the standard normal distribution;

[0138] from Recover from The ideal variance It can be expressed by the following formula:

[0139] ,

[0140] In the reverse diffusion process of generating hyperspectral images, the denoising process can be expressed by the following formula:

[0141] ,

[0142] The reverse diffusion mean ,variance The value of is determined by the vector , Diffusion step index and land feature classification map , Multispectral Imagery Predict through Unet network;

[0143] (1-6) Execution The denoising operation generates the result , the denoising process of each step can be expressed by the following formula:

[0144] ,

[0145] in The value is less than greater than 0, Represents the Unet network.

[0146] The fourth step is to build a recurrent dual-task reciprocal learning framework including segmentation and synthesis:

[0147] The specific steps are as follows:

[0148] (1) A recurrent dual-task reciprocal learning framework is established by constructing a segmentation network based on the fusion of hyperspectral and lidar data and a hyperspectral synthesis network based on the object classification map and multispectral imagery. The results of the segmentation network are used as the input of the synthesis network, and the results of the synthesis network are also used as the network of the segmentation network.

[0149] (2) For the segmentation network, the network input is a hyperspectral image and LiDAR images , the network output is a land feature classification map ; Its segmentation loss function is designed as follows:

[0150] (2-1) For paired hyperspectral images and LiDAR images There are corresponding land feature classification map labels data, through the cross entropy loss in supervised learning The training segmentation network parameters can be expressed by the following formula:

[0151] ,

[0152] in Represents the cross entropy loss calculation process, represents the segmentation network;

[0153] (3) For the synthetic network, the network input is the ground object classification map , Multispectral Imagery , the network output is a hyperspectral image ; Its synthetic loss function is designed as follows:

[0154] (3-1) For paired feature classification maps , Multispectral Imagery Has its corresponding hyperspectral image label data, through the mean squared error loss in supervised learning The training segmentation network parameters can be expressed by the following formula:

[0155] ,

[0156] in Represents the mean square error loss calculation process, represents a synthetic network;

[0157] (4) A cycle is defined as a forward propagation of the segmentation network and the synthesis network in sequence, where the result of the segmentation network is used as the input of the synthesis network, and the result of the synthesis network is also used as the input of the segmentation network; for data with missing modalities, that is, data missing in the segmentation network , which is missing in the synthetic network The consistency loss is calculated by two cycles:

[0158] (4-1) For the lack of hyperspectral images in the segmentation network , and there are paired lidar images and feature classification map labels The data is first forward propagated through the synthetic network to generate a synthetic hyperspectral image. , and then generate through a forward propagation of the segmentation network ; Repeat the above cycle again to generate , calculate its consistency loss , which can be expressed by the following formula:

[0159] ,

[0160] in Represents the mean square error loss calculation process;

[0161] (4-2) For the lack of object classification map in the synthetic network , and there are paired multispectral images and hyperspectral imagery tags The data is first segmented and forward propagated to generate a classification map of the ground objects. , and then generate through a forward propagation of the synthetic network ; Repeat the above cycle again to generate , calculate its consistency loss , which can be expressed by the following formula:

[0162] ,

[0163] in Represents the mean square error loss calculation process.

[0164] The fifth step is to train the multimodal image segmentation network based on cyclic dual-task reciprocal learning and obtain the results:

[0165] The specific steps are as follows:

[0166] (1) By , , and Perform back propagation, where , is used to update the model parameters of the segmentation network, , Used to update the model parameters of the synthesis network;

[0167] (2) Determine whether the set number of training rounds has been reached. If so, complete the training of the multimodal image segmentation network based on cyclic dual-task reciprocal learning; otherwise, continue training;

[0168] (3) The hyperspectral image and lidar image are spliced ​​according to the channel dimension, and the final segmentation result image is obtained through the segmentation network.

[0169] A remote sensing multimodal image segmentation method based on cyclic dual-task reciprocal learning proposed by this method significantly improves the detail retention of the segmentation results and the consistency of cross-modal features, and effectively enhances the adaptability of the network in the absence of modality. Compared with the existing remote sensing image segmentation technology, the present invention uses a cyclic dual-task reciprocal learning framework to promote each other between the segmentation network and the synthesis network, solving the problem of decreased segmentation accuracy caused by incomplete information and modality misalignment. In addition, this method gives full play to the advantages of hyperspectral and lidar data in multimodal data fusion, and optimizes the information flow of feature maps by maintaining attention jump connections based on Fourier transform structures, further improving the processing power and robustness of the model. The present invention also combines the training paradigms of supervised learning and semi-supervised learning, reduces the dependence on a large amount of labeled data, and reduces the cost of manual labeling. This method shows higher efficiency and stability in remote sensing image segmentation, and has broad application prospects and practical value.

[0170] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A remote sensing multimodal image segmentation method based on cyclic dual-task reciprocal learning, characterized in that: The following steps are involved: 11) Construct multimodal remote sensing image segmentation task dataset and synthesis task dataset: obtain paired hyperspectral images h and lidar images l to construct segmentation task dataset; obtain paired ground object classification maps y and multispectral images m to construct synthesis task dataset; further perform data augmentation on the segmentation task dataset and synthesis task dataset to expand the dataset; 12) Construct a segmentation network based on hyperspectral and lidar data fusion: Construct a segmentation network with the input as the paired augmented hyperspectral image h segaug , augmented LiDAR image segaug A segmentation network comprising an encoder and a decoder; 13) Construct a hyperspectral synthesis network based on the object classification map and multispectral imagery: Construct a hyperspectral synthesis network with the augmented object classification map y as input synaug , augmented multispectral image m synaug A hyperspectral synthesis network, in which the model adopts a conditional diffusion generation model; 14) Construct a reciprocal dual-task learning framework including segmentation and synthesis: Construct a reciprocal dual-task learning framework for image segmentation and hyperspectral synthesis, where the segmentation results serve as the input of the synthesis network and the synthesis results serve as the input of the segmentation network; 15) Training a multimodal image segmentation network based on cyclic dual-task reciprocal learning and obtaining results: Train the remote sensing multimodal image segmentation network through a training paradigm that combines supervised learning and semi-supervised learning, and obtain the final test results.

2. According to claim 1, a remote sensing multimodal image segmentation method based on cyclic dual-task reciprocal learning is characterized in that: The construction of a segmentation network based on hyperspectral and lidar data fusion includes the following steps: 21) In the segmentation task dataset, for the corresponding ground object classification map label The data is then labeled with its hyperspectral image h, lidar image l and ground feature classification map Splice together by channel dimension; 211) According to the size of 32×32, the spliced ​​hyperspectral image h, lidar image l and ground feature classification map label Divide into multiple image blocks of corresponding sizes; 212) For each segmented image block p seg , randomly select an image block p from the remaining multiple image blocks seg ′, p seg ′, replace the image block p with a sub-image block of a random size selected from seg The sub-image block at the corresponding position in ; 213) divides all image blocks processed by 1112) into corresponding augmented hyperspectral images h according to the channel dimension segaug , augmented LiDAR image segaug and augmented feature classification map labels As a segmentation task dataset after data augmentation; 22) In the synthetic task dataset, for the corresponding hyperspectral image labels The data is used to classify the object y, multispectral image m and hyperspectral image labels Splice together by channel dimension; 221) According to the size of 32×32, the spliced ​​ground feature classification map y, multispectral image m and hyperspectral image label Divide into multiple image blocks of corresponding sizes; 222) For each segmented image block p syn , randomly select an image block p from the remaining multiple image blocks syn ′, p syn ′, and replace p with a sub-image block of random size syn The sub-image block at the corresponding position in ; 223) All image blocks processed by 1122) are divided into corresponding augmented object classification images y according to the channel dimension synaug , augmented multispectral image m synaug and augmented hyperspectral imagery labels As a synthetic task dataset after data augmentation.

3. The remote sensing multimodal image segmentation method based on cyclic dual-task reciprocal learning according to claim 1, characterized in that: The construction of a segmentation network based on hyperspectral and lidar data fusion includes the following steps: 31) A segmentation network based on hyperspectral and lidar data fusion includes an encoder, a decoder and a bottleneck layer, and the encoder and the decoder are connected through a Fourier transform-based structure to maintain an attention jump connection structure; 311) The encoder of the segmentation network is set to include 4 consecutive downsampling structures: the downsampling structure is a convolution layer with a convolution kernel of 3×3, a batch normalization layer, a Relu layer, a convolution layer with a convolution kernel of 3×3, a batch normalization layer, a Relu layer stack, and finally a 2x maximum pooling layer; the paired augmented hyperspectral image h segaug , augmented LiDAR image segaug As input, they are spliced ​​together according to the channel dimension, and four outputs of different sizes, e1, e2, e3, and e4, are obtained through four consecutive downsampling structures. 312) Set the bottleneck layer of the segmentation network: the bottleneck layer receives the output e4 of the fourth downsampling structure as input, and the structure is a fully connected layer, a Gelu layer, a fully connected layer, and a Gelu layer; the output of the bottleneck layer is b; 313) The decoder of the segmentation network is set to include 4 consecutive upsampling structures: the first upsampling structure receives the output b of the bottleneck layer as input, and the upsampling structure is a convolution layer with a convolution kernel of 3×3, a batch normalization layer, a Relu layer, a convolution layer with a convolution kernel of 3×3, a batch normalization layer, and a Relu layer stack, and finally a transposed convolution with a step size of 2 is performed to complete the upsampling operation; after four consecutive upsampling structures, four outputs of different sizes d1, d2, d3, and d4 are obtained respectively; 314) Set a Fourier transform-based structure-preserving attention jump connection structure connecting the encoder and the decoder: the four downsampling structures of the encoder respectively obtain outputs of four different sizes, and the output of each size passes through the structure-preserving attention jump connection structure; assuming that the feature map is x, the Fourier transform process can be expressed by the following formula: |F(a,b)|=Fourier(x), Where Fourier() stands for Fourier transform, represents phase, |F()| represents amplitude, a and b represent the horizontal and vertical frequency coordinates in the frequency domain; The phase represents the structural information of the feature map. It is element-wise multiplied with a tensor t of the same size that can be gradient descent, and then inverse Fourier transformed into the feature map x′. The process can be expressed by the following formula: Where Inverse Fourier() represents the inverse Fourier transform, Represents element-wise multiplication: The outputs e1, e2, e3, and e4 of the four downsampling structures are connected through the structure-preserving attention jump structure to obtain four feature maps a1, a2, a3, and a4 respectively. a4 is concatenated to the first output b of the bottleneck layer, a3 is spliced ​​to the first output d1 of the decoder, a2 is spliced ​​to the second output d2 of the decoder, a1 is spliced ​​to the third output d3 of the decoder; The feature map d4 obtained by the fourth upsampling structure is input into the tanh activation function; 32) The hyperspectral data and lidar data are spliced ​​in the channel dimension and input into the segmentation network based on the fusion of hyperspectral and lidar data to obtain the final prediction map of the land object classification results.

4. The remote sensing multimodal image segmentation method based on cyclic dual-task reciprocal learning according to claim 1, characterized in that: The construction of a hyperspectral synthesis network based on a ground object classification map and multispectral images comprises the following steps: 41) A hyperspectral synthesis network based on ground object classification maps and multispectral images uses a conditional diffusion generation model as a synthesis framework: 411) The network input is the augmented feature classification map y synaug and augmented multispectral image m synaug , and its corresponding augmented hyperspectral image label is 412) Construct a standard Unet network to predict the mean of the multidimensional Gaussian distribution in the denoising process; 413) In the forward process of diffusion from hyperspectral image to noise, let the maximum number of steps of adding noise be K, then h (k) represents the feature map obtained by forward diffusion of the hyperspectral image after k steps, where k is a positive integer less than K; (k-1) h is sampled from the normal distribution of (k) The forward diffusion process representing one step can be expressed by the following formula; where β (k) ∈{β (1) , β (2) , ..., β (K) } is uniformly distributed between 0.0001 and 0.02, representing the intensity of noise added at each step; represents a normal distribution with mean μ and variance σ; I represents the identity matrix; represents a standard normal distribution; 414) Randomly select step number k from h (0) The vector h is generated after k steps of noise addition operation (k) , which can be expressed by the following formula: where α (k) =1-β (k) , and β (k) ∈{β (1) , β (2) , …, β (K) } follows a uniform distribution between 0.0001 and 0.02, ε (k) Obey standard normal distribution; 415) from h (k) Recover from h (k-1) The ideal mean It can be expressed by the following formula: where ε (k) It obeys the standard normal distribution; From h (k) Recover from h (k-1) The ideal variance It can be expressed by the following formula: In the reverse diffusion process of generating hyperspectral images, the denoising process can be expressed by the following formula: The reverse diffusion mean variance The value of is determined by the vector h (k) , diffusion step index k and augmented feature classification map y synaug , augmented multispectral image m synaug Predict through Unet network; 416) Perform K-step denoising operation to generate result h (0) , the denoising process of each step can be expressed by the following formula: The value of k is less than K and greater than 0, and Unet() represents the Unet network.

5. The remote sensing multimodal image segmentation method based on cyclic dual-task reciprocal learning according to claim 1, characterized in that: The construction of a recurrent dual-task reciprocal learning framework including segmentation and synthesis includes the following steps: 51) A recurrent dual-task reciprocal learning framework is established by constructing a segmentation network based on hyperspectral and lidar data fusion and a hyperspectral synthesis network based on object classification maps and multispectral images; the results of the segmentation network are used as the input of the synthesis network, and the results of the synthesis network are also used as the input of the segmentation network; 52) For the segmentation network, the network input is the augmented hyperspectral image h segaug and augmented LiDAR images segaug , the network output is the object classification map y segout ; Its segmentation loss function is designed as follows: 521) For the paired augmented hyperspectral images h segaug and augmented LiDAR images segaug There is a corresponding augmented feature classification map label Data, through the cross entropy loss in supervised learning I CE The training segmentation network parameters can be expressed by the following formula: Where cross entropy() represents the cross entropy loss calculation process; 53) For the synthetic network, the network input is the augmented object classification map y synaug , augmented multispectral image m synaug , the network output is a hyperspectral image h synout ; Its synthetic loss function is designed as follows: 531) For the paired augmented feature classification map y synaug , augmented multispectral image m synaug Has its corresponding augmented hyperspectral image label The data is obtained by using the mean square error loss L in supervised learning. MSE The training segmentation network parameters can be expressed by the following formula: Where MSE() represents the mean square error loss calculation process; 54) One cycle is defined as the segmentation network and the synthesis network performing a forward propagation successively, in which the result of the segmentation network is used as the input of the synthesis network, and the result of the synthesis network is also used as the input of the segmentation network; for data with missing modalities, that is, the augmented hyperspectral image h is missing in the segmentation network segaug , the synthetic network lacks the augmented object classification map y synaug The consistency loss is calculated by two cycles: 541) For the lack of augmented hyperspectral image h in the segmentation network segaug , and there are pairs of augmented LiDAR images l segaug and augmented feature classification map labels The data is first forward propagated through the synthetic network to generate a synthetic hyperspectral image. Then, the forward propagation of the segmentation network is repeated to generate y′; the above cycle is repeated again to generate y″, and its consistency loss L is calculated. SC , which can be expressed by the following formula: L SC =MSE(y′,y″), Where MSE() represents the mean square error loss calculation process; 542) For the lack of augmented feature classification map y in the synthetic network synaug , and there are paired augmented multispectral images m synaug and augmented hyperspectral imagery labels The data is first segmented and forward propagated to generate a classification map of the ground objects. Then, the synthetic network is forward propagated once more to generate h′; the above cycle is repeated once more to generate h″, and its consistency loss L is calculated. AC , which can be expressed by the following formula: L AC =MSE(h′,h″), Where MSE() represents the mean square error loss calculation process.

6. The remote sensing multimodal image segmentation method based on cyclic dual-task reciprocal learning according to claim 1, characterized in that: The training of the multimodal image segmentation network based on cyclic dual-task reciprocal learning and the result acquisition include the following steps: 61) By L CE , L MSE , L SC and L SC Perform back propagation, where L CE , L SC is used to update the model parameters of the segmentation network, L MSE , L AC Used to update the model parameters of the synthesis network; 62) Determine whether the set number of training rounds has been reached, if so, complete the training of the multimodal image segmentation network based on cyclic dual-task reciprocal learning, otherwise continue the training; 63) The hyperspectral image and lidar image are spliced ​​according to the channel dimension, and the final segmentation result image is obtained through the segmentation network.

Images