LiDAR data and optical image change detection method based on multiple tasks

Abstract

The invention provides a multi-task-based LiDAR data and optical image change detection method, belongs to the technical field of multi-modal remote sensing image processing, and specifically comprises an image generation task and a multi-modal change detection task. The relation between a high-resolution optical image and an nDSM is mined through a diffusion model, a high-quality optical-nDSM is stably generated, elevation information is supplemented for the high-resolution optical image to assist in detection of building changes, a multi-modal change detection model based on improved U2-Net is constructed, multi-level and multi-scale feature extraction is achieved, and the detection accuracy is improved. And a multi-modal feature enhancement module is combined to effectively enhance the key region of each level of depth feature. The method has the beneficial effects that the relation between the high-resolution optical image and the nDSM is fully mined, the conversion between the high-resolution optical image and the nDSM is learned from an unchanged region according to the basic principle of a diffusion model, and a multi-modal feature enhancement module is designed to effectively enhance a feature key region, so that the change detection precision is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal remote sensing image processing, and in particular relates to a change detection method for cities based on multi-task LiDAR data and optical images. Background Art

[0002] Urban change detection can use images of the same geographical area at different times to obtain a binary change map to mark whether each pixel has changed. It is widely used in urban research, such as urban land planning and monitoring, disaster monitoring and assessment, urban map database update and many other fields. LiDAR data (Light Detection and Ranging, LiDAR) and high-resolution optical image (High Resolution Optical Image, HRI) are often used for urban change detection. They each have their own advantages in urban change detection tasks. LiDAR data can reflect the relative ground height information and material properties of objects, can distinguish objects with similar appearance but different materials and heights, and is easy to detect building changes, but lacks texture details, is expensive to obtain, and has a long imaging cycle. High-resolution optical images can provide spectral information and texture details of surface objects, have price advantages and are easy to obtain, but lack elevation information. LiDAR data can solve the problem of high-resolution optical images lacking elevation information, and high-resolution optical images can solve the problem of LiDAR data lacking spectral details and long imaging cycle. By mining the connection between LiDAR data and optical images, optical-LiDAR data that retains image details and structures is generated for optical images, and elevation information is added to optical images, which can solve the problem of difficult detection of some building change types. Therefore, it is of great significance to use the characteristics of LiDAR data and high-resolution optical images and the connection between them to accurately detect urban land use change maps.

[0003] With the rise of deep learning, many supervised semantic segmentation architectures have been introduced to process remote sensing images of different modalities before and after changes. Existing studies usually use generative adversarial networks (GANs) to explore the connection between images before and after changes, reduce the modal difference between images before and after changes by learning the transformation of images before and after changes, convert the images before (after) changes to the modalities of images after (before) changes, and then use supervised semantic segmentation models to process the converted images to detect changes. Optical images that lack elevation information are converted to LiDAR data modalities with elevation information, thereby converting the LiDAR data-optical image change detection task into a LiDAR data-LiDAR data change detection task, which helps detect building changes. However, GANs often require adversarial training and careful selection of hyperparameters and regularization functions, which makes them unstable and has the problem of mode collapse. In addition, the LiDAR data reconstructed by GANs usually have unnatural artifacts, which significantly affects the accuracy of downstream change detection tasks.

[0004] In recent years, denoising diffusion probalistic models (DDPM) have been widely studied and applied due to their powerful image generation capabilities and stable training process, and have become one of the most mainstream technologies in the field of image generation. Compared with GAN, diffusion models have a more stable training process and can generate higher quality and more diverse images that retain details and structures. Diffusion models with powerful image generation capabilities can be used to explore the connection between high-resolution optical images and nDSM. By learning the conversion from high-resolution optical images to nDSM in complex urban scenes, higher quality optical-nDSM can be generated for optical images. However, high-resolution optical images, LiDAR data, and generated optical-nDSM belong to different modalities. The modal differences lead to unsatisfactory change information extraction effects of simple feature splicing or addition. Therefore, how to mine, integrate and enhance the key information before and after the change of different modalities is the key to multimodal change detection. The attention mechanism has the ability to filter irrelevant or low-quality areas, identify and concentrate on key areas, and has been widely studied and applied. It has become one of the technologies that have attracted much attention in the field of semantic segmentation. In order to enhance the key areas of different modal features to obtain a more complete change picture, it is necessary to effectively integrate the information before and after the change of different modalities. In summary, a multimodal change detection framework is needed that can generate high-quality optical-nDSM for high-resolution optical images and effectively enhance the features of different modalities. Summary of the invention

[0005] In view of the problem that existing multimodal change detection methods cannot stably generate high-quality optical-nDSM and effectively enhance multimodal features, the method of the present invention provides a change detection method based on multi-task LiDAR data and optical images, which includes an image generation task and a multimodal change detection task. In the image generation task, a diffusion model is used to complete the image generation task, which can be specifically divided into a forward diffusion process and a reverse denoising process. The forward diffusion process gradually adds Gaussian noise to the image, and the reverse denoising process gradually reconstructs the image from the Gaussian noise image. The diffusion model learns the transformation between paired high-resolution optical images and nDSM samples from the unchanged areas of the two to generate high-quality optical-nDSM-assisted optical images that retain the high-resolution optical image content to detect building changes. In the multimodal change detection task, a method based on the improved U 2 -Net's multimodal change detection model, which is based on an encoding-decoding structure with three encoder branches and one decoder, where each encoding block and decoding block consists of a U-shaped residual block (Residual U-block, RSU). An effective multimodal feature enhancement module is designed after the encoding stage to enhance the key feature areas. The multimodal feature enhancement module integrates local and global contextual information from different levels of features in different modalities to effectively enhance the key areas of deep features in different modalities. Among them, the training process of the diffusion model and the multimodal change detection model is independent.

[0006] The proposed method for urban change detection based on multi-task LiDAR data and optical images has the following two significant features: the first significant feature is that it can mine the connection between high-resolution optical images and nDSM. The trained diffusion model can reconstruct the optical-nDSM corresponding to the optical image according to the learned conversion from high-resolution optical image to nDSM when there is only high-resolution optical image, and supplement the elevation information for the optical feature; the second significant feature is that it has a powerful adaptive multimodal feature enhancement capability. The multimodal feature enhancement module can integrate local and global context information from different levels of features of different modalities, and effectively enhance the key areas of different modal features. The proposed method for urban change detection based on multi-task LiDAR data and optical images has the following advantages: the model can fully and stably learn the conversion from high-resolution optical image to nDSM, generate high-quality optical-nDSM with image content retained for high-resolution optical image, and effectively capture and utilize the local and global context information of feature maps before and after different modal changes, so as to better complete the multimodal urban change detection task.

[0007] The present invention provides a method for detecting urban changes based on LiDAR data and optical images using multi-tasks, and the implementation steps are as follows:

[0008] Step 1: Obtain original LiDAR data, high-resolution optical images and real labels of land use changes, construct a multimodal change detection dataset, and then obtain a training set for the diffusion model. The diffusion model includes a forward diffusion process and a reverse denoising process. The forward diffusion process gradually adds Gaussian noise to the image, and the reverse denoising process gradually reconstructs the image from the Gaussian noise image;

[0009] Step 2: Use the training samples in the training set of the diffusion model to train the diffusion model;

[0010] Step 3: Generate an optical-nDSM that retains the high-resolution optical image content using the trained diffusion model;

[0011] Step 4: Train a multimodal change detection model using LiDAR data, high-resolution optical images, the optical-nDSM generated in step 3, and the true labels of the changes. The multimodal change detection model includes three encoders, a decoder, and a multimodal feature enhancement module. Each encoder and decoder is composed of a U-shaped residual block. The multimodal feature enhancement module is used to enhance the key feature areas of the image.

[0012] Step 5: Use the trained multimodal change detection model to predict the land use change map.

[0013] Furthermore, in step 1, the specific process of constructing a multimodal change detection dataset and then obtaining a diffusion model training set is as follows:

[0014] Step 1.1: Obtain the original LiDAR data and high-resolution optical images of a certain area. The two types of data are obtained at different times, that is, the data before and after the change respectively;

[0015] Step 1.2: Use ArcMap software to integrate the original LiDAR data and convert it into raster format to obtain digital surface model (DSM) raster data, digital elevation model (DEM) raster data and echo intensity raster data;

[0016] Step 1.3: Calculate the difference between DSM data and DEM data to obtain normalized Digital Surface Model (nDSM) raster data, and obtain LiDAR data based on nDSM raster data and echo intensity raster data. nDSM raster data and echo intensity raster data together constitute two characteristic bands of LiDAR data;

[0017] Step 1.4: Use Envi software to combine LiDAR data and high-resolution remote sensing images to outline the real labels of land use changes through visual interpretation. The types of changes include changes in buildings, parking lots, roads, grasslands and other land features, with building changes being the main focus.

[0018] Step 1.5: Crop the LiDAR data, high-resolution optical images, and true change labels into image blocks of preset sizes to obtain a multimodal change detection dataset, and randomly divide the multimodal change detection dataset into a training set and a test set;

[0019] Step 1.6: Randomly select some training samples from the training set obtained in step 1.5 to train the diffusion model, and convert the pixel values ​​of the changed areas of the high-resolution optical image and nDSM to 0 according to the change labels to construct the training set of the diffusion model.

[0020] Furthermore, in step 2, the specific process of training the diffusion model is:

[0021] Step 2.1: Set the hyperparameters of the diffusion model, including learning rate, batch size, total number of forward noise steps, and the relationship between noise and time steps, and randomly initialize these hyperparameters;

[0022] Step 2.2: Load a high-resolution optical image (HRI) of a preset size that masks the change region from the diffusion model training set mask and nDSM raster data that masks the changed areas mask , randomly extract time steps t∈[1,T] and random noise is a Gaussian distribution; the forward diffusion process of the diffusion model includes a total of T time steps, from time step 1 to time step T, the input nDSM is gradually mask Add Gaussian noise; the forward diffusion process q is defined as follows:

[0023]

[0024] Among them, x0 is the input nDSM raster data nDSM that masks the changed area mask , x t is the nDSM Gaussian noise image at time step t obtained according to the forward diffusion rule, x t-1 is the nDSM Gaussian noise image at time step t-1 obtained according to the forward diffusion process, β t is the noise adding scheme, I is the unit matrix, α t : =1-β t , High-resolution optical imagery (HRI) that masks the changing regions when training the diffusion model mask 、x tand time step t, which is the input of the noise prediction model U-Net;

[0025] Step 2.3: Train the noise prediction model U-Net to predict the noise from x in the forward diffusion process under high-resolution optical imaging conditions. t-1 to x t The added noise, that is, the goal of training U-Net is to learn the inverse process of each step in the forward diffusion process; calculate the real noise ∈ added by the forward diffusion process and the noise ∈ predicted by U-Net θ The error between simple As the loss function of the noise prediction model U-Net:

[0026]

[0027] Among them, C HRI For high-resolution optical images, when training the diffusion model, C HRI The value is HRI mask , express expectations;

[0028] Step 2.4: Randomly select time step t in the range [1, T], repeat step 2.3, and train U-Net to learn to predict the value of x under the condition of high-resolution optical image. t-1 to x t Noise is added until U-Net converges.

[0029] Furthermore, in step 3, the specific process of using the trained diffusion model to generate an optical-nDSM that retains the image content is as follows:

[0030] Step 3.1: Load the original high-resolution optical image of the unmasked change area from the dataset and input it into the diffusion model network;

[0031] Step 3.2: The reverse denoising process of the diffusion model is:

[0032]

[0033] Among them, C HRI For high-resolution optical images, when generating optical-nDSM, C HRI The value of is the original high-resolution optical image without masking the changed area. According to the trained U-Net and the above formula, under the condition of the original high-resolution optical image, denoising is performed step by step from time step T to time step 0 to obtain the generated optical-nDSM.

[0034] Furthermore, in step 4, when training the multimodal change detection model, it is necessary to extract deep features of different modal data at different levels. The specific process is as follows:

[0035] The LiDAR data, high-resolution optical images, and the optical-nDSM generated by the diffusion model are input into a change detection network with three encoders, namely the LiDAR branch, the optical branch, and the optical-nDSM branch, which are used to extract LiDAR depth features. Optical depth features and optical-nDSM depth features Where i is the number of stages, i.e., i = [1, 2, 3, 4, 5]. If the original input feature size is C×H×W, where C, H, and W represent the number of channels, height, and width of the image, respectively, then the output feature sizes of the five stages are In order to capture more local and global information from the feature map, a tiny U-shaped residual block is used as the encoder and decoder. By mixing receptive fields of different scales, contextual information of different scales is effectively captured to extract multi-level and multi-scale feature representations of the image.

[0036] Furthermore, after the encoding stage, the extracted LiDAR depth features, optical depth features and optical-nDSM depth features at all levels are enhanced using the Multimodal Feature Enhancement Module (MFEM) to improve the ability of the multimodal change detection model to detect the changed area, integrate the local and global context information from the features at all levels of different modalities, and enhance the key areas of the features at all levels, specifically:

[0037] The multimodal feature enhancement module first performs deep feature extraction on each level of LiDAR. and optical depth features Perform channel attention weighting to obtain and

[0038]

[0039] Where i = [1, 2, 3, 4, 5], AvgPool and MaxPool represent global average pooling and global maximum pooling respectively, MLP represents the fully connected operation, Sigmoid represents the Sigmoid activation function, and ⊙ represents the channel dimension multiplication.

[0040] Afterwards, and Upsample to restore Resolution, get and After global average pooling and 1×1 convolution operations, and then splicing, the spatial feature map is obtained. and

[0041]

[0042]

[0043] Among them, Up represents the upsampling operation, Cat represents the feature concatenation operation, GloAvg represents the global average pooling, and Conv 1×1 Represents a 1×1 convolution operation.

[0044] In particular, in order to capture information from different receptive fields, the deepest features and After upsampling, it is restored to After the resolution is reached, global average pooling, 1×1 convolution operation and 4 3×3 convolution operations with different expansion rates are performed to obtain the deepest spatial feature map. and

[0045]

[0046] in, represents a dilated convolution operation with a convolution kernel of 3 and a dilation rate of d. Characterizes the optical branch, Represent the features of the LiDAR branch respectively.

[0047] Then, the five spatial feature maps obtained by the optical branch and the LiDAR branch are concatenated to obtain the global spatial feature map GS HRI and GS LiDAR , and then integrate the local and global spatial context information before and after the different modal changes through splicing and convolution operations to obtain Spatial attention map AM of the same resolution:

[0048]

[0049] AM=Sigmoid(Conv 3×3 (LeakyReLU(Bn(Conv 3×3 (Cat(GS HRI ,GS LiDAR ))))))

[0050] Among them, LeakyReLU represents the LeakyReLU activation function, Bn represents batch normalization, Conv 3×3 represents a 3×3 convolution operation, Represent the spatial feature maps of the first, second, third, and fourth layers of the optical branch, respectively. Respectively represent the spatial feature maps of the first, second, third, and fourth layers of the LiDAR branch, Represents the spatial feature map of the fifth layer of the optical branch, Represents the spatial feature map of the fifth layer of the LiDAR branch.

[0051] Spatial Attention Map AM and and optical-nDSM depth features Perform element-wise multiplication to enhance the key areas of the feature and obtain the enhanced LiDAR depth feature Optical depth features and optical-nDSM depth features

[0052]

[0053] in represents spatial dimension multiplication, and down represents downsampling.

[0054] Finally, the enhanced features are concatenated to obtain the outputs of the multimodal feature enhancement module at five different resolutions:

[0055]

[0056] Furthermore, in step 4, after obtaining the five features of different resolutions output by the multimodal feature enhancement module, they are input into the decoder for decoding, specifically:

[0057] Will Feed it to the decoder for decoding step by step to get the output of each level of decoder Then, the four features F with different resolutions output by the decoder are integrated i To obtain a more complete change map, the three intermediate features of different resolutions are upsampled to restore the original input image resolution and 2×2 convolution operations, and the last layer of features F after convolution operation is combined. 1 Perform element-by-element addition and then perform a 2×2 convolution operation to obtain the change graph, as follows:

[0058]

[0059] Among them, Conv 2×2 Represents a 2×2 convolution operation with an output channel number of 2, Up 2×2 represents a 2×2 transposed convolution operation with a stride of 2, F 4 、F 3 、F 2 、F 1 They represent the features of the output after the convolution operation, They represent the output results of the decoder after decoding step by step, and Change map represents the change map.

[0060] Furthermore, in step 4, the loss function of the multimodal change detection model is:

[0061] The loss between the change map output by the model and the true label of land use change is calculated. In view of the extreme imbalance of positive and negative samples in the dataset, the binary cross entropy loss L is used. bce and the dice coefficient loss L dice The combined mixed loss of is used as the loss function of the multimodal change detection network. The loss function Loss is defined as follows:

[0062]

[0063] Loss = L bce +L dice

[0064] where y i represents the true label value of the i-th pixel, Represents the predicted value of the i-th pixel, y i =0 means that the pixel is unchanged, y i =1 indicates that the pixel is a change, and N represents the total number of pixels.

[0065] Furthermore, the specific process of predicting the land use change map in step 5 is as follows:

[0066] Step 5.1: Load the multimodal change detection model and test set image patches trained in step 4.

[0067] Step 5.2: Input the test set LiDAR data, high-resolution optical images, and optical-nDSM generated by the diffusion model into the multimodal change detection model to extract deep features of various levels containing rich local and global information.

[0068] Step 5.3: Filter and enhance the deep features at each level through the multimodal feature enhancement module to obtain the enhanced multimodal features at each level.

[0069] Step 5.4: Feed the enhanced multimodal features of each level obtained in step 5.3 to the decoder for decoding, restore the feature map resolution to the input image resolution step by step, integrate the features of four different resolutions output by the decoder, and obtain the change map.

[0070] A storage medium having a computer program stored thereon, characterized in that: when the computer program is executed by a processor, the steps of the multi-task based LiDAR data and optical image change detection method are implemented.

[0071] The method of the present invention has the following significant effects:

[0072] 1. The present invention uses a diffusion model to fully learn the conversion of high-resolution optical images to nDSM from the unchanged areas of training samples and explore the connection between the two, so as to generate a high-quality optical-nDSM that retains details and structures for optical images. By generating an optical-nDSM for a high-resolution optical image, the problem of lack of elevation information in the optical image is solved. The generated optical-nDSM can effectively assist high-resolution optical images in detecting building changes in urban scenes, especially changes from parking lots to buildings with similar spectra to parking lots.

[0073] 2. The present invention utilizes the improved U 2 -Net's multimodal change detection model processes images before and after multimodal changes. First, it uses a U-shaped residual block to perform multi-level and multi-scale feature extraction on the original LiDAR data, high-resolution optical images and the generated optical-nDSM. Then, the multimodal feature enhancement module integrates the local and global contextual information of LiDAR depth features and optical depth features to obtain a spatial attention map containing rich local and global information, effectively enhancing the key areas of depth features of different modalities at all levels, thereby effectively improving the model's ability to detect changed areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 is a flow chart of the present invention;

[0075] Figure 2 is a training sample of the diffusion model in Example 1 of the present invention and its original data, nDSM (a) masking the change area, high-resolution optical image (b) masking the change area, original nDSM (c), original high-resolution optical image (d) and change label (e);

[0076] Figure 3 This is the basic principle of the diffusion model in step 2 and step 3 of embodiment 1 of the present invention;

[0077] Figure 4 is a schematic diagram of a multimodal change detection model in step 4 and step 5 of embodiment 1 of the present invention;

[0078] Figure 5 is a schematic diagram of a multimodal feature enhancement module (MFEM) according to Embodiment 1 of the present invention;

[0079] Figure 6 This is the variation test result of Example 1 of the present invention. DETAILED DESCRIPTION

[0080] The outstanding features and remarkable improvements of the present invention are further illustrated by the following examples, which are intended only to illustrate the present invention but in no way to limit the present invention.

[0081] like Figure 1 As shown, the present invention provides a change detection method based on multi-task LiDAR data and optical images, and the implementation steps are as follows:

[0082] Step 1: Obtain original LiDAR data, high-resolution optical images and real labels of land use changes, construct a multimodal change detection dataset, and then obtain a training set for the diffusion model. The diffusion model includes a forward diffusion process and a reverse denoising process. The forward diffusion process gradually adds Gaussian noise to the image, and the reverse denoising process gradually reconstructs the image from the Gaussian noise image;

[0083] Step 2: Use the training samples in the training set of the diffusion model to train the diffusion model;

[0084] Step 3: Generate an optical-nDSM that retains the high-resolution optical image content using the trained diffusion model;

[0085] Step 4: Train a multimodal change detection model using LiDAR data, high-resolution optical images, the optical-nDSM generated in step 3, and the true labels of the changes. The multimodal change detection model includes three encoders, a decoder, and a multimodal feature enhancement module. Each encoder and decoder is composed of a U-shaped residual block. The multimodal feature enhancement module is used to enhance the key feature areas of the image.

[0086] Step 5: Use the trained multimodal change detection model to predict the land use change map.

[0087] Example 1

[0088] (I) Obtain LiDAR data, high-resolution optical images, and real labels of land use changes to construct the Buffalo dataset.

[0089] Step 1.1: Obtain the original LiDAR data with a resolution of 1 meter in part of the urban area of ​​Buffalo, Erie County, Pennsylvania, USA in 2008, and the high-resolution optical image with a resolution of 0.3048 meters in the area in 2021. The two types of data are obtained at different times, that is, the data before and after the change. The optical image contains 4 bands, namely the near-infrared band, the red band, the green band, and the blue band.

[0090] Step 1.2: Use ArcMap software to integrate the original LiDAR data and convert it into raster format to obtain digital surface model (DSM) raster data, digital elevation model (DEM) raster data and echo intensity raster data.

[0091] Step 1.3: Use ArcMap software to downsample the high-resolution optical image to the same resolution as the LiDAR data. Calculate the difference between the DSM data and the DEM data to obtain the nDSM grid data, and obtain the LiDAR data based on the nDSM grid data and the echo intensity grid data. The nDSM grid data and the echo intensity grid data together constitute the two characteristic bands of the LiDAR data.

[0092] Step 1.4: Use Envi software combined with high-resolution optical images and LiDAR data to outline the true labels of land use changes through visual interpretation. The types of land use changes include changes in buildings, parking lots, roads, and grasslands, with building changes being the main type. Changed pixels are marked as 1, and unchanged pixels are marked as 0.

[0093] Step 1.5: Crop the Erie area data without overlap and randomly divide the sub-images to make the Buffalo dataset. Randomly divide the multimodal change detection dataset into a training set and a test set, that is, the Buffalo dataset contains a training set and a test set. Both the training set and the test set can be divided into LiDAR data, high-resolution optical images, and true labels of changes. The image size is 256 × 256. Among them, the training set contains 62 sub-images, and the test set contains 16 sub-images. The ratio of the training set to the test set is about 8:2.

[0094] Step 1.6: In order to ensure that the multimodal change detection model in the downstream change detection task has a good ability to suppress pseudo-changes, only some training samples are randomly extracted from the training set obtained in step 1.5 for training the diffusion model. In this embodiment, 50 training samples are randomly extracted from the training set for training the diffusion model, of which the high-resolution optical image, nDSM and the true label of the change will be used to train the diffusion model. According to the change label, the pixel values ​​of the high-resolution optical image and the nDSM change area are converted to 0 to construct the training set of the diffusion model. A training sample of the diffusion model is as follows: Figure 2 As shown, Figure 2 (a) is the nDSM that masks the change area. Figure 2 (b) is a high-resolution optical image that masks the changed area. Figure 2 (c) in the figure is the original nDSM. Figure 2 (d) is the original high-resolution optical image. Figure 2 (e) in the figure is a change label.

[0095] (ii) Training the diffusion model to learn the conversion from high-resolution optical images to nDSM. The specific process is as follows:

[0096] Step 2.1: Set the hyperparameters of the diffusion model, including learning rate, batch size, total number of forward noise addition steps, relationship between noise and time step, etc. In this embodiment, the batch size is set to 4, the learning rate is set to 0.0001, the total number of forward noise addition steps T of the diffusion model is set to 1000, and the relationship between noise and time t is set to a linear relationship.

[0097] Step 2.2: Load the 256×256 size high-resolution optical image HRI that masks the change area from the training set mask and nDSM raster data that masks the changed areas mask , randomly sample time steps t∈[1,1000] and random noise is a Gaussian distribution. The forward diffusion process q is defined as follows:

[0098]

[0099] Among them, x0 is the input nDSM raster data nDSM that masks the changed area mask , x t is the nDSM Gaussian noise image at time step t obtained according to the forward diffusion rule, x t-1 is the nDSM Gaussian noise image at time step t-1 obtained according to the forward diffusion process, β t is the noise adding scheme, I is the unit matrix, α t : =1-β t , High-resolution optical images (HRI) that mask the changing regions when training the diffusion model mask 、x t and time step t, which is the input of the noise prediction model U-Net.

[0100] Step 2.3: Train the noise prediction model U-Net to predict the noise from x in the forward diffusion process under high-resolution optical imaging conditions. t-1 to x t The added noise, that is, the goal of training U-Net is to learn the inverse process of each step in the forward diffusion process; among them, U-Net is the noise prediction model in the reverse denoising process of the diffusion model, and is a deep neural network used to predict noise. The real noise ∈ added in the forward diffusion process and the noise ∈ predicted by U-Net are θ The error between simple As the loss function of the noise prediction model U-Net:

[0101]

[0102] Among them, C HRI For high-resolution optical images, when training the diffusion model, C HRIThe value is HRI mask .

[0103] Step 2.4: Randomly select a time step t in the range [1,1000] and repeat step 2.3 to train U-Net to learn to predict the value from x under the condition of high-resolution optical image. t-1 to x t Noise is added until U-Net converges.

[0104] (iii) Generate optical-nDSM for all high-resolution optical images of the Buffalo dataset.

[0105] Step 3.1: Load the original high-resolution optical image and input it into the diffusion model network.

[0106] Step 3.2: Reverse denoising process according to the diffusion model:

[0107]

[0108] Among them, C HRI For high-resolution optical images, when generating optical-nDSM, C HRI The value of is the original high-resolution optical image without masking the changed area. According to the trained U-Net and the above formula, under the condition of the original high-resolution optical image, denoising is performed step by step from time step 1000 to time step 0 to obtain the generated optical-nDSM. Figure 3 This is a schematic diagram of the forward denoising process and the reverse denoising process of the diffusion model, which shows the basic principle of the diffusion model. In the figure, Forward / DiffusionProcess represents the forward diffusion process of the diffusion model, and Reverse / Denoise Process represents the reverse denoising process. t |x t-1 ) represents the forward diffusion process x t-1 Under the condition x t distribution.

[0109] (iv) Train a multimodal change detection model using LiDAR data, high-resolution optical images, the optical-nDSM generated in step 3, and land use change labels.

[0110] Step 4.1: Set the hyperparameters of the multimodal change detection model, such as setting the initial learning rate to 0.1, the learning rate adjustment strategy to Linear, the batch size to 4, etc., and randomly initialize the model parameters.

[0111] Step 4.2: Input the LiDAR data, high-resolution optical imagery, and the optical-nDSM generated by the diffusion model into a change detection network with three encoders to extract LiDAR depth features respectively. Optical depth features and optical-nDSM depth features Where i is the number of stages, i.e., i = [1, 2, 3, 4, 5]. A tiny U-shaped residual block is used as the encoder and decoder to effectively capture contextual information of different scales by mixing receptive fields of different scales and extract multi-level and multi-scale feature representations. Figure 4 This is a schematic diagram of the multimodal change detection model. Diffusion Model represents the diffusion model, Encoder represents the encoder, Decoder represents the decoder, and Multimodal Feature Enhancement Module represents the multimodal feature enhancement module. The three encoder branches do not share weights.

[0112] Step 4.3: Use the Multimodal Feature Enhancement Module to enhance the extracted LiDAR depth features, optical depth features, and optical-nDSM depth features at all levels to improve the ability of the multimodal change detection model to detect changed areas. It is used to integrate local and global contextual information from features at all levels of different modalities and enhance the key areas of features at all levels. Figure 5 This is a schematic diagram of a multimodal feature enhancement module. Specifically:

[0113] The multimodal feature enhancement module first performs deep feature extraction on each level of LiDAR. and optical depth features Perform channel attention weighting to obtain and

[0114]

[0115] Where i = [1, 2, 3, 4, 5], AvgPool and MaxPool represent global average pooling and global maximum pooling respectively, MLP represents the fully connected operation, Sig represents the Sigmoid activation function, and ⊙ represents the channel dimension multiplication;

[0116] Afterwards, and Upsample to restore Resolution, get and After global average pooling and 1×1 convolution operations, and then splicing, the spatial feature map is obtained. and

[0117]

[0118] Among them, Up represents up-sampling operation (Up-sampling), Cat (Concatenation) represents feature concatenation operation, GloAvg represents global average pooling, Conv 1×1 Represents a 1×1 convolution operation;

[0119] In order to capture information from different receptive fields, the deepest features and After upsampling, it is restored to After the resolution is reached, global average pooling, 1×1 convolution operation and 4 3×3 convolution operations with different expansion rates are performed to obtain the deepest spatial feature map. and

[0120]

[0121] in, represents a dilated convolution operation with a kernel of 3 and a dilation rate of d;

[0122] Then, the five spatial feature maps obtained by the optical branch and the LiDAR branch are concatenated to obtain the global spatial feature map GS HRI and GS LiDAR , and then integrate the local and global spatial context information before and after the different modal changes through splicing and convolution operations to obtain Spatial attention map AM (Attention Map) of the same resolution:

[0123]

[0124] AM=Sig(Conv 3×3 (LeakyReLU(Bn(Conv 3×3 (Cat(GS HRI ,GS LiDAR )))))

[0125] Among them, LeakyReLU represents the LeakyReLU activation function, Bn (BatchNorm) represents batch normalization, Conv 3×3 Represents a 3×3 convolution operation;

[0126] Spatial Attention Map AM and and optical-nDSM depth features Perform element-wise multiplication to enhance the key areas of the feature and obtain the enhanced LiDAR depth feature Optical depth features and optical-nDSM depth features

[0127]

[0128] in Represents spatial dimension multiplication, and dowm (Down-sampling) represents down sampling;

[0129] Finally, the enhanced features are concatenated to obtain the outputs of the multimodal feature enhancement module at five different resolutions:

[0130]

[0131] Step 4.4: The obtained in step 4.3 Feed it to the decoder for decoding step by step to get the output of each level of decoder Then, a more complete change map is obtained by integrating the four features of different resolutions output by the decoder. The three intermediate features of different resolutions are upsampled to restore to the original input image resolution and 2×2 convolution operation, and then combined with the last layer feature F after convolution operation. 1 Perform element-by-element addition and then perform a 2×2 convolution operation to obtain the change graph, as follows:

[0132]

[0133]

[0134] Among them, Conv 2×2 Represents a 2×2 convolution operation with an output channel number of 2, Up 2×2 represents a 2×2 transposed convolution operation with a stride of 2, F 4 、F 3 、F 2 、F 1 They represent the features of the output after the convolution operation, They represent the output results of the decoder after decoding step by step, and Change map represents the change map.

[0135] Step 4.5: Calculate the loss between the change map output by the model and the true label of land use change, using binary cross entropy loss L bce and the dice coefficient loss L dice The combined mixed loss is used as the loss function. Repeat steps 4.2, 4.3, and 4.4 to iteratively optimize the model weights and save the optimal model.

[0136] Finally, the specific process of step 5 predicting the test set change graph is as follows:

[0137] Step 5.1: Load the multimodal change detection model and test set image patches trained in step 4.

[0138] Step 5.2: Input the test set LiDAR data, high-resolution optical images, and optical-nDSM generated by the diffusion model into the multimodal change detection model to extract deep features of various levels containing rich local and global information.

[0139] Step 5.3: Screen and enhance different modal features through the multimodal feature enhancement module.

[0140] Step 5.4: Feed the enhanced features of each level obtained in step 5.3 to the decoder for decoding, restore the feature map resolution to the input image resolution step by step, integrate the features of four different resolutions output by the decoder, and obtain the change map. Figure 6 The change test results in the test set are shown, (a) is the echo intensity data obtained in 2008, (b) is the nDSM obtained in 2008, (c) is the high-resolution optical image obtained in 2021, (d) is the true label of the change, and (e) is the binary change map; black indicates no change and white indicates change.

[0141] A storage medium having a computer program stored thereon, characterized in that: when the computer program is executed by a processor, the steps of the multi-task based LiDAR data and optical image urban change detection method are implemented.

[0142] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

[0143] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

Claims

1. A change detection method based on multi-task LiDAR data and optical images, characterized by: The following steps are involved: Step 1: Obtain original LiDAR data, high-resolution optical images and real labels of land use changes, construct a multimodal change detection dataset, and then obtain a training set for the diffusion model. The diffusion model includes a forward diffusion process and a reverse denoising process. The forward diffusion process gradually adds Gaussian noise to the image, and the reverse denoising process gradually reconstructs the image from the Gaussian noise image; Step 2: Use the training samples in the training set of the diffusion model to train the diffusion model; Step 3: Generate an optical-nDSM that retains the high-resolution optical image content using the trained diffusion model; Step 4: Train a multimodal change detection model using LiDAR data, high-resolution optical images, the optical-nDSM generated in step 3, and the true labels of the changes. The multimodal change detection model includes three encoders, a decoder, and a multimodal feature enhancement module. Each encoder and decoder is composed of a U-shaped residual block. The multimodal feature enhancement module is used to enhance the key feature areas of the image. Step 5: Use the trained multimodal change detection model to predict the land use change map.

2. The method for detecting changes in LiDAR data and optical images based on multi-tasks according to claim 1, characterized in that: In step 1, the specific process of constructing a multimodal change detection dataset and then obtaining a diffusion model training set is as follows: Step 1.1: Obtain the original LiDAR data and high-resolution optical images of a certain area. The two types of data are obtained at different times, that is, the data before and after the change respectively; Step 1.2: Use ArcMap software to integrate the original LiDAR data and convert it into raster format to obtain DSM raster data, DEM raster data and echo intensity raster data; Step 1.3: Calculate the difference between DSM data and DEM data to obtain nDSM raster data. The nDSM raster data and echo intensity raster data together constitute the two characteristic bands of LiDAR data. Step 1.4: Use Envi software to combine LiDAR data and high-resolution remote sensing images to outline the true labels of land use changes through visual interpretation. The types of land use changes include changes in buildings, parking lots, roads, and grasslands; Step 1.5: Crop the LiDAR data, high-resolution optical images, and true change labels into image blocks of preset sizes to obtain a multimodal change detection dataset, and randomly divide the multimodal change detection dataset into a training set and a test set; Step 1.6: Randomly select some training samples from the training set obtained in step 1.5 to train the diffusion model, and convert the pixel values ​​of the changed areas of the high-resolution optical image and nDSM to 0 according to the true label of the change to construct the training set of the diffusion model.

3. The method for detecting changes in LiDAR data and optical images based on multi-tasks according to claim 1, characterized in that: In step 2, the specific process of training the diffusion model is: Step 2.1: Set the hyperparameters of the diffusion model, including learning rate, batch size, total number of forward noise steps, and the relationship between noise and time steps, and randomly initialize these hyperparameters; Step 2.2: Load a high-resolution optical image (HRI) of a preset size that masks the change region from the diffusion model training set mask and nDSM raster data that masks the changed areas mask , randomly extract time steps t∈[1,T] and random noise is a Gaussian distribution; the forward diffusion process of the diffusion model includes a total of T time steps, from time step 1 to time step T, the input nDSM is gradually mask Add Gaussian noise; the forward diffusion process q is defined as follows: Among them, x0 is the input nDSM raster data nDSM that masks the changed area mask , x t is the nDSM Gaussian noise image at time step t obtained according to the forward diffusion process, x t-1 is the nDSM Gaussian noise image at time step t-1 obtained according to the forward diffusion process, β t is the noise adding scheme, I is the unit matrix, α t : =1-β t , High-resolution optical images (HRI) that mask the changing regions when training the diffusion model mask 、x t and time step t, which is the input of the noise prediction model U-Net; Step 2.3: Train the noise prediction model U-Net to predict the noise from x in the forward diffusion process under high-resolution optical imaging conditions. t-1 to x t The added noise, that is, the goal of training U-Net is to learn the inverse process of each step in the forward diffusion process; the real noise ∈ added by the forward diffusion process and the noise ∈ predicted by U-Net θ The error between simple As the loss function of the noise prediction model U-Net: Among them, C HRI For high-resolution optical images, when training the diffusion model, C HRI The value is HRI mask ; Step 2.4: Randomly select time step t in the range [1, T], repeat step 2.3, and train U-Net to learn to predict the value of x under the condition of high-resolution optical image. t-1 to x t Noise is added until U-Net converges.

4. The method for detecting changes in LiDAR data and optical images based on multi-tasks according to claim 3, characterized in that: In step 3, the specific process of using the trained diffusion model to generate an optical-nDSM that retains the image content is as follows: Step 3.1: Load the original high-resolution optical image of the unmasked change area from the multimodal change detection dataset and input it into the diffusion model; Step 3.2: The reverse denoising process of the diffusion model is: Among them, C HRI For high-resolution optical images, when generating optical-nDSM, C HRI The value of is the original high-resolution optical image without masking the changed area. According to the trained U-Net and the above formula, under the condition of the original high-resolution optical image, denoising is performed step by step from time step T to time step 0 to obtain the generated optical-nDSM.

5. The method for detecting changes in LiDAR data and optical images based on multi-tasks according to claim 1, characterized in that: In step 4, when training the multimodal change detection model, it is necessary to extract the deep features of different modal data at all levels through three encoders. The specific process is as follows: The LiDAR data, high-resolution optical images, and the optical-nDSM generated by the diffusion model are input into a change detection network with three encoders, namely the LiDAR branch, the optical branch, and the optical-nDSM branch, which are used to extract LiDAR depth features. Optical depth features and optical-nDSM depth features Where i is the number of stages, i = [1, 2, 3, 4, 5]; if the original input feature size is C×H×W, C, H, and W represent the number of channels, height, and width of the image, respectively, then the output feature sizes of the five stages are In order to capture more local and global information from the feature map, U-shaped residual blocks are used as encoders and decoders to effectively capture contextual information of different scales by mixing receptive fields of different scales and extract multi-level and multi-scale feature representations of the image.

6. The method for detecting changes in LiDAR data and optical images based on multi-tasks according to claim 5, characterized in that: In step 4, after the encoding stage, the extracted LiDAR depth features, optical depth features, and optical-nDSM depth features at all levels are enhanced using the multimodal feature enhancement module to improve the ability of the multimodal change detection model to detect the changed area. It is used to integrate the local and global context information from the features at all levels of different modalities and enhance the key areas of the features at all levels, specifically: The multimodal feature enhancement module first performs deep feature extraction on each level of LiDAR. and optical depth features Perform channel attention weighting to obtain and Where i = [1, 2, 3, 4, 5], AvgPool and MaxPool represent global average pooling and global maximum pooling respectively, MLP represents the fully connected operation, Sig represents the Sigmoid activation function, and ⊙ represents the channel dimension multiplication; Afterwards, and Upsample to restore Resolution, get and After global average pooling and 1×1 convolution operations, and then splicing, the spatial feature map is obtained. and Among them, Up represents the upsampling operation, Cat represents the feature concatenation operation, GloAvg represents the global average pooling, and Conv 1×1 Represents a 1×1 convolution operation; In order to capture information from different receptive fields, the deepest features and After upsampling, it is restored to After the resolution is reached, global average pooling, 1×1 convolution operation and 4 3×3 convolution operations with different expansion rates are performed to obtain the deepest spatial feature map. and in, represents a dilated convolution operation with a kernel of 3 and a dilation rate of d; Characterizes the optical branch, Represents the features of the LiDAR branch; Then, the five spatial feature maps obtained by the optical branch and the LiDAR branch are spliced ​​together to obtain the global spatial feature map GS HRI and GS LiDAR , and then integrate the local and global spatial context information before and after the different modal changes through splicing and convolution operations to obtain Spatial attention map AM of the same resolution: AM=Sig(Conv 3×3 (LeakyReLU(Bn(Conv 3×3 (Cat(GS HRI ,GSL iDAR ))))) Among them, LeakyReLU represents the LeakyReLU activation function, Bn represents batch normalization, Conv 3×3 Represents a 3×3 convolution operation; Represent the spatial feature maps of the first, second, third, and fourth layers of the optical branch, respectively. Respectively represent the spatial feature maps of the first, second, third, and fourth layers of the LiDAR branch, Represents the spatial feature map of the fifth layer of the optical branch, Represents the spatial feature map of the fifth layer of the LiDAR branch; Spatial Attention Map AM and and optical-nDSM depth features Perform element-wise multiplication to enhance the key areas of the feature and obtain the enhanced LiDAR depth feature Optical depth features and optical-nDSM depth features in represents spatial dimension multiplication, and down represents downsampling; Finally, the enhanced features are concatenated to obtain the outputs of the multimodal feature enhancement module at five different resolutions:

7. The method for detecting changes in LiDAR data and optical images based on multi-tasks according to claim 6, characterized in that: In step 4, after obtaining the five features of different resolutions output by the multimodal feature enhancement module, they are input into the decoder for decoding, specifically: Will Feed it to the decoder for decoding step by step to obtain the output after decoding at each level Then, the four features F with different resolutions output by the decoder are integrated i To obtain a more complete change map, the three intermediate features of different resolutions are upsampled to restore to the original input image resolution and 2×2 convolution operation, and the last layer of features F after convolution operation is combined. 1 Perform element-by-element addition and then perform a 2×2 convolution operation to obtain the change graph, as follows: Among them, Conv 2×2 Represents a 2×2 convolution operation with an output channel number of 2, Up 2×2 represents a 2×2 transposed convolution operation with a stride of 2, F 4 、F 3 、F 2 、F 1 They represent the features of the output after the convolution operation, They represent the output results of the decoder after decoding step by step, and Change map represents the change map.

8. The method for detecting changes in LiDAR data and optical images based on multi-tasks according to claim 1, characterized in that: In step 4, the loss function of the multimodal change detection model is: To solve the problem of extremely unbalanced positive and negative samples in the data set, a binary cross entropy loss L is used. bce and the dice coefficient loss L dice The combined mixed loss of is used as the loss function of the multimodal change detection network; the loss function loss is defined as follows: loss=L bce +L dice where y i represents the true label value of the i-th pixel, Represents the predicted value of the i-th pixel, y i =0 means that the pixel is unchanged, y i =1 indicates that the pixel is a change, and N represents the total number of pixels.

9. The method for detecting changes in LiDAR data and optical images based on multi-tasks according to claim 1, characterized in that: Step 5 specifically includes the following steps: Step 5.1: Load the trained multimodal change detection model and test set image patches; Step 5.2: Input the test set LiDAR data, high-resolution optical images, and optical-nDSM generated by the diffusion model into the multimodal change detection model to extract deep features of various levels containing rich local and global information; Step 5.3: Filter and enhance the deep features at each level through the multimodal feature enhancement module to obtain the enhanced multimodal features at each level; Step 5.4: Feed the enhanced multimodal features of each level obtained in step 5.3 to the decoder for decoding, restore the feature map resolution to the input image resolution step by step, integrate the features of four different resolutions output by the decoder, and obtain the land use change map.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the change detection method based on multi-task LiDAR data and optical images described in any one of claims 1 to 9 are implemented.

Images