A LiDAR prior-guided satellite image DEM generation method, medium and device

Abstract

The present invention proposes a method, medium and device for generating satellite image DEM guided by LiDAR prior, including: randomly cutting out two adjacent blocks with a certain overlap area from the whole DSM data; using the matching satellite image DSM data and LiDAR DEM data to train the conditional latent diffusion model, and adaptively extracting the corresponding terrain prior features from the two adjacent DSM blocks respectively; adopting the feature fusion strategy based on the cross attention mechanism, hierarchically embedding the generated terrain prior features into the Transformer regression network, and constructing a multivariate integrated loss function to explicitly guide the DSM blocks to enhance the details of DEM prediction; constructing a geometric consistency loss function to explicitly punish the scale deviation that may exist in the regression process of the adjacent area DEM, and forcing the model to output a geometrically consistent adjacent area DEM. The present invention can not only realize the selective highlighting of terrain-related features, but also generate a globally consistent large-scale seamless DEM.

Description

Technical Field

[0001] The invention belongs to the field of aerospace photogrammetry data processing, and specifically relates to a LiDAR a priori guided satellite image DEM generation method, medium and equipment. Background Art

[0002] Optical satellite stereo observation can obtain remote sensing data with continuous spatial coverage of any area in orbit, which is an important means of global geographic information resource construction and forest resource survey. It is of great practical significance to use high-resolution satellite images to efficiently produce global / regional digital terrain products (such as DSM, DEM, etc.) and forest height data.

[0003] At present, the production process of DSM (Digital Surface Model) based on satellite stereo images is relatively mature, but how to automatically generate DEM (Digital Elevation Model) data using satellite images still puzzles the international academic community. As a classic problem in the field of geospatial data processing, automatic extraction of DEM has been widely studied. Early research focused on DEM generation methods based on ground filtering, and recent research has gradually begun to explore DEM generation methods based on ground regression.

[0004] The DEM generation method based on ground filtering usually includes two steps: ground filtering and elevation interpolation. Such methods have been widely used in LiDAR (Light Detection and Ranging) point clouds, but they are difficult to robustly process satellite photogrammetry data due to the defects of satellite image DSM. With the rapid development of generative AI, some recent studies have begun to explore DEM generation methods based on ground regression. Such methods attempt to directly realize the conversion from DSM to DEM in a unified deep learning framework. However, due to the single strategy adopted and the lack of reasonable constraints, the generated DEM is relatively smooth and difficult to directly seamlessly splice. Therefore, there is an urgent need to explore new ideas and methods for the automatic generation of DEM from high-resolution satellite images. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a method, medium and device for generating satellite image DEM guided by LiDAR prior. By decomposing the satellite image DEM generation problem into a combination of three relatively simple problems (prior extraction, DEM regression and consistency optimization), the complexity of the entire problem can be effectively reduced, thereby achieving robust and high-quality generation of satellite image DEM.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for generating satellite image DEM guided by LiDAR priori, characterized by comprising:

[0008] Step 1: Randomly cut out two adjacent DSM blocks with a certain overlap area from the entire DSM data;

[0009] Step 2: Establish a conditional latent diffusion model and use the supporting satellite image DSM data and LiDAR DEM data for training to adaptively extract the corresponding terrain prior features from two adjacent DSM blocks;

[0010] Step 3: Establish a DEM regression model. The DEM regression model adopts a feature fusion strategy based on a cross-attention mechanism to hierarchically embed the extracted terrain prior features into the Transformer regression network to obtain enhanced DEM regression features, and perform DEM prediction based on them; construct a multivariate integrated loss function to train the DEM regression model;

[0011] Step 4: Build a DEM regression optimization framework based on the DEM regression model, and construct a geometric consistency loss function to train the DEM regression optimization framework, which is used to explicitly penalize the scale deviation that may exist in the regression process and force the DEM regression model to output geometrically consistent DEMs of adjacent areas;

[0012] Step 5: Repeat steps 1-4 until the sum of all DSM blocks covers the original entire DSM data, and generate a large-scale seamless DEM with global geometric consistency based on the DEM prediction results of all DSM blocks.

[0013] To optimize the above technical solutions, the specific measures taken also include:

[0014] Further, in step 2, the conditional latent diffusion model is composed of a latent encoder e, a latent encoder s, a conditional denoising diffusion probability model and a latent decoder;

[0015] The latent encoder e encodes the combined data of the satellite image DSM data and the LiDAR DEM data into a latent feature Z; the latent encoder s encodes the satellite image DSM data alone into a latent feature C; the conditional denoising diffusion probability model takes the latent feature Z as the starting point of the forward diffusion process, and gradually adds Gaussian noise to it in T iterations to obtain Z T , given Z T Under the conditions of and C, the denoising network is used to gradually transform Gaussian noise into prior features through T iterations. The latent decoder takes the prior features Decodes into a DEM corresponding to the input DSM tile.

[0016] Furthermore, in step 2, the terrain prior features are extracted as follows: the latent encoder s in the trained conditional latent diffusion model and the denoising network of the conditional denoising diffusion probability model are combined as an excellent terrain prior generation model for adaptively extracting corresponding terrain prior features from two adjacent DSM blocks.

[0017] Furthermore, in step 3, the DEM regression model obtains enhanced DEM regression features in the following manner:

[0018] Pass two adjacent DSM blocks to a series of Transformer layers to extract DSM local features;

[0019] A feature fusion strategy based on the cross-attention mechanism is adopted to fuse the extracted terrain prior features and DSM local features at multiple scales to obtain enhanced DEM regression features.

[0020] Furthermore, in step 3, the multivariate ensemble loss function is specifically as follows:

[0021]

[0022] L all =α1l1+α2l grad +α3l norm

[0023] In the formula, l1 represents the L1 norm loss, l grad represents the gradient loss, l norm Denotes the normal vector loss, L all is the multivariate integrated loss function, which represents the total loss used for DEM regression model training; d e represents the DEM prediction value, d r represents the true value of LiDAR DEM, is the component in the x direction, is the component in the y direction, n is the total number of pixels, represent represent α1, α2 and α3 are the weights of each sub-loss; using the multivariate integrated loss function L all And the labeled training set is used to train the DEM regression model.

[0024] Furthermore, in step 4, the geometric consistency loss function is specifically as follows:

[0025]

[0026] Where, L Geo represents the geometric consistency loss function, Indicates Ea Zhong and E b There are overlapping parts. Indicates E b Zhong and E a There is an overlap, V stands for E a With E b The overlapping area, E a With E b The DEMs corresponding to two adjacent DSM blocks generated by the DEM regression model prediction.

[0027] Furthermore, in step 4, the training of the DEM regression optimization framework is specifically as follows: using the geometric consistency loss function L Geo And the labeled training set is used to train the DEM regression optimization framework.

[0028] Furthermore, in step 5, in the process of generating a globally geometrically consistent large-scale seamless DEM based on the DEM prediction results of all DSM blocks, the overlapping areas in the DEM prediction results of all DSM blocks are averaged and fused.

[0029] Correspondingly, the present invention also proposes a computer-readable storage medium storing a computer program, characterized in that the computer program enables a computer to execute the above-mentioned LiDAR a priori guided satellite image DEM generation method.

[0030] Correspondingly, the present invention also proposes an electronic device, characterized in that it includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the LiDAR a priori guided satellite image DEM generation method as described above is implemented.

[0031] The beneficial effects of the present invention are:

[0032] (1) This paper combines the feature fusion strategy based on the cross-attention mechanism and adopts a hierarchical approach to efficiently embed the terrain prior features generated by the diffusion model into the Transformer-based DEM regression model, which can not only achieve the selective highlighting of terrain-related features, but also enhance the generalization ability of the model to handle complex scenes;

[0033] (2) In the process of adjacent area DEM regression, the present invention pre-introduces geometric consistency loss to explicitly penalize possible adjacent area prediction deviations and forces the model to output results of the same scale, which can not only promote the generation of globally consistent large-scale seamless DEM, but also improve the prediction accuracy of the DEM regression model. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of the overall process framework of the method of the present invention.

[0035] Figure 2 Schematic diagram of a potential encoder based on deformable convolution in the present invention.

[0036] Figure 3 Schematic diagram of the conditional potential diffusion model in the present invention.

[0037] Figure 4 Schematic diagram of the Transformer regression network with multi-level prior embedding in the present invention.

[0038] Figure 5 Schematic diagram of the feature fusion module based on the cross-attention mechanism in the present invention.

[0039] Figure 6 Schematic diagram of the adjacent area DEM regression optimization framework with geometric consistency constraints in the present invention. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0041] In one embodiment, the present invention proposes a method for generating satellite image DEM guided by LiDAR priori, such as Figure 1 As shown, the method mainly includes the following steps.

[0042] Step 1: Randomly cut out two adjacent blocks with a certain overlap in the entire DSM data.

[0043] Step 2: Use the satellite image DSM data and airborne LiDAR DEM data to train the deep generative model, and adaptively extract the corresponding excellent terrain priors from two adjacent DSM blocks. It includes the following sub-steps:

[0044] Step 2.1: Collect a large number of satellite image DSM blocks with a side length of 256 pixels and the corresponding airborne LiDARDEM blocks.

[0045] Step 2.2: Construct a conditional latent diffusion model, such as Figure 3 As shown in Figure 2, the model consists of two latent encoders (based on deformable convolution, such as Figure 2 The conditional denoising diffusion probability model first takes feature Z as the starting point of its forward diffusion process and gradually adds Gaussian noise to it in T iterations to obtain Z. T; Then at a given Z T Under the conditions of and C, the denoising network is used to gradually transform Gaussian noise into prior features through T iterations. The potential decoder is responsible for converting Decodes into a DEM corresponding to the input DSM tile.

[0046] Step 2.3: Use the data collected in step 2.1 to perform supervised training on the model constructed in step 2.2.

[0047] Step 2.4: Combine the trained latent encoder s in step 2.2 and the neural network layers of the denoising network module in the conditional denoising diffusion probability model into a good terrain prior generation model, and adaptively generate / extract the corresponding good terrain prior from two adjacent DSM blocks.

[0048] Step 3: Adopt the feature fusion strategy based on the cross-attention mechanism, hierarchically embed the generated terrain prior features into the Transformer regression network, and construct a multivariate integrated loss function to explicitly guide the detail enhancement of DSM slicing to DEM prediction. It includes the following sub-steps:

[0049] Step 3.1: Pass two adjacent DSM blocks to a series of Transformer layers to extract deep DSM local features.

[0050] Step 3.2: Using the feature fusion strategy based on the cross-attention mechanism, the terrain prior features extracted in step 2.4 and the DSM local features are fused at multiple scales to obtain enhanced DEM regression features, such as Figure 5 shown.

[0051] Step 3.3: Combine the neural network layers used in steps 3.1 to 3.2, and add several convolutional layers and regression output layers to construct Figure 4 DEM regression model shown.

[0052] Step 3.4, define a multivariate ensemble loss function as follows:

[0053]

[0054] L all =α1l1+α2l graa +α3l norm

[0055] In the formula, l1 represents the L1 norm loss, l grad represents the gradient loss, l norm Denotes the normal vector loss, L all represents the total loss for DEM regression model training; d erepresents the DEM prediction value, d r represents the true value of LiDAR DEM, is the component in the x direction, is the component in the y direction, n is the total number of pixels, represent represent α1, α2 and α3 are the weights of each sub-loss. Using the total loss function L all And the labeled training set Figure 4 The DEM regression model shown is trained.

[0056] Step 4: Construct a geometric consistency loss function to explicitly penalize the scale deviation that may exist in the regression process of the adjacent area DEM, and force the model to output geometrically consistent adjacent area DEM. It includes the following sub-steps:

[0057] Step 4.1: Use the DEM regression model trained in step 3 to predict and generate the DEMs corresponding to two adjacent DSM blocks.

[0058] Step 4.2: Define a geometric consistency loss function as follows:

[0059]

[0060] In the formula, Indicates E a Zhong and E b There are overlapping parts. Indicates E b Zhong and E a There is an overlap, V stands for E a With E b Using the geometric consistency loss function L Geo And the labeled training set Figure 6 The neighboring area DEM regression optimization framework is trained as shown.

[0061] Step 4.3: Use the trained DEM regression optimization framework to process the DSM blocks of two adjacent areas to obtain geometrically consistent DEM prediction results for adjacent areas.

[0062] Step 5: Repeat steps 1 to 4 until the sum of all DSM slices covers the original complete DSM, and average the overlapping areas in the DEM prediction results of all DSM slices to finally generate a large-scale seamless DEM with global geometric consistency.

[0063] In another embodiment, the present invention proposes a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the LiDAR a priori guided satellite image DEM generation method as described in the first embodiment.

[0064] In another embodiment, the present invention proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the LiDAR a priori guided satellite image DEM generation method as described in Example 1 is implemented.

[0065] In the embodiments disclosed in the present application, the computer storage medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. The computer storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of computer storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0066] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0067] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A LiDAR prior-guided satellite image DEM generation method, characterized in that: include: Step 1: Randomly cut out two adjacent DSM blocks with overlapping areas from the entire DSM data; Step 2: Establish a conditional latent diffusion model and use the matching satellite image DSM data and LiDAR DEM data for training, so as to adaptively extract the corresponding terrain prior features from two adjacent DSM blocks; in step 2, the conditional latent diffusion model is composed of a latent encoder e, a latent encoder s, a conditional denoising diffusion probability model and a latent decoder; the latent encoder e encodes the combined data of the matching satellite image DSM data and LiDAR DEM data into a latent feature Z; the latent encoder s encodes the satellite image DSM data alone into a latent feature C; The conditional denoising diffusion probability model takes the potential feature Z as the starting point of the forward diffusion process and gradually adds Gaussian noise to it in T iterations to obtain Z T , given Z T Under the conditions of and C, the denoising network is used to gradually transform Gaussian noise into prior features through T iterations. The latent decoder takes the prior features Decode into DEM corresponding to the input DSM slice; Step 3: Establish a DEM regression model. The DEM regression model adopts a feature fusion strategy based on a cross-attention mechanism to hierarchically embed the extracted terrain prior features into the Transformer regression network to obtain enhanced DEM regression features, and perform DEM prediction based on them; construct a multivariate integrated loss function to train the DEM regression model; Step 4: Build a DEM regression optimization framework based on the DEM regression model, and construct a geometric consistency loss function to train the DEM regression optimization framework, which is used to explicitly penalize the scale deviation in the regression process and force the DEM regression model to output geometrically consistent DEMs of adjacent areas. Step 5: Repeat steps 1-4 until the sum of all DSM blocks covers the original entire DSM data, and generate a large-scale seamless DEM with global geometric consistency based on the DEM prediction results of all DSM blocks.

2. The method for generating satellite image DEM using LiDAR priori guidance according to claim 1, characterized in that: In step 2, the terrain prior features are extracted as follows: the latent encoder s in the trained conditional latent diffusion model and the denoising network of the conditional denoising diffusion probability model are combined as an excellent terrain prior generation model for adaptively extracting corresponding terrain prior features from two adjacent DSM blocks.

3. The method for generating satellite image DEM guided by LiDAR a priori as claimed in claim 1, characterized in that: In step 3, the DEM regression model obtains enhanced DEM regression features in the following manner: Pass two adjacent DSM blocks to a series of Transformer layers to extract DSM local features; A feature fusion strategy based on the cross-attention mechanism is adopted to fuse the extracted terrain prior features and DSM local features at multiple scales to obtain enhanced DEM regression features.

4. The method for generating satellite image DEM guided by LiDAR a priori as claimed in claim 3, characterized in that: In step 3, the multivariate ensemble loss function is specifically as follows: L all ＝α1l1+α2l grad +α3l norm In the formula, l1 represents the L1 norm loss, l grad represents the gradient loss, l norm Denotes the normal vector loss, L all is the multivariate integrated loss function, which represents the total loss used for DEM regression model training; d e represents the DEM predicted value, d r represents the true value of LiDAR DEM, is the component in the x direction, is the component in the y direction, n is the total number of pixels, represent represent α1, α2 and α3 are the weights of each sub-loss; using the multivariate integrated loss function L all And the labeled training set is used to train the DEM regression model.

5. The method for generating satellite image DEM guided by LiDAR a priori as claimed in claim 1, characterized in that: In step 4, the geometric consistency loss function is as follows: Where, L Geo represents the geometric consistency loss function, Indicates E a Zhong and E b There are overlapping parts. Indicates E b Zhong and E a There is an overlap, V stands for E a With E b The overlapping area, E a With E b The DEMs corresponding to two adjacent DSM blocks generated by the DEM regression model prediction.

6. The method for generating satellite image DEM guided by LiDAR a priori as claimed in claim 5, characterized in that: In step 4, the training of the DEM regression optimization framework is specifically as follows: using the geometric consistency loss function L Geo And the labeled training set is used to train the DEM regression optimization framework.

7. The method for generating satellite image DEM guided by LiDAR a priori as claimed in claim 1, characterized in that: In step 5, in the process of generating a globally geometrically consistent large-scale seamless DEM based on the DEM prediction results of all DSM blocks, the overlapping areas in the DEM prediction results of all DSM blocks are averaged and fused.

8. A computer-readable storage medium storing a computer program, characterized in that: The computer program enables a computer to execute the LiDAR a priori guided satellite image DEM generation method as described in any one of claims 1 to 7.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for generating satellite image DEM guided by a priori LiDAR is implemented.

Images