Method, device and equipment for generating online map based on diffusion model and medium

Abstract

The application relates to a remote sensing image generation online map method, device, equipment and medium, a plurality of different transformed sample images are obtained by performing geometric transformation on remote sensing sample images in each group of sample image pairs, the plurality of transformed sample images are added to the corresponding sample image pairs to obtain a training data set, each training data set is used to train an online map generation model to obtain a trained online map generation model, an encoder in a perception image compression network is used to map remote sensing sample images and corresponding transformed sample images from a pixel space to a feature space, a forward diffusion and reverse denoising are performed on the feature space by a denoising diffusion bridge network, and then an output of the denoising diffusion bridge network is mapped from the feature space to the pixel space by a decoder in the perception image compression network, and a real-time remote sensing image is input into the trained online map generation model to obtain a real-time network map. By adopting the method, a map with clearer boundaries and brighter colors can be generated in real time.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image conversion, in particular to a remote sensing image generation online map method and device based on a diffusion model, equipment and medium. BACKGROUND

[0002] Due to the adaptability and low cost of remote sensing images, the generation of online maps based on remote sensing images has been widely used. Before the development of deep learning, traditional map making methods require manual acquisition of vector data, which is time-consuming and laborious. In particular, in the event of sudden events such as fires, earthquakes, and tsunamis, map updates may not be timely enough. Therefore, it is of great significance to develop a method that can quickly generate accurate maps in emergency situations.

[0003] Some existing researches use the idea of domain mapping to realize end-to-end conversion of remote sensing images to maps (RSMT). Although these methods have significant effects and high efficiency in online map generation, the generated online maps still have problems of feature distortion and inaccurate boundaries to some extent. SUMMARY

[0004] Therefore, it is necessary to provide a remote sensing image generation online map method, device, equipment and medium based on a diffusion model, which can generate accurate maps with high efficiency.

[0005] A remote sensing image generation online map method based on a diffusion model, the method comprising:

[0006] Obtaining a sample image data set, the sample image data set comprising a plurality of groups of sample image pairs, each group of sample image pairs comprising a remote sensing sample image and a corresponding network map;

[0007] Performing geometric transformation on the remote sensing sample image in each group of sample image pairs to obtain a plurality of different transformed sample images, adding the plurality of transformed sample images to the corresponding sample image pair to obtain a training data group, and constructing a training data set according to a plurality of training data groups;

[0008] Training the online map generation model using the training data set to obtain a trained online map generation model, the online map generation model comprising a perception image compression network and a denoising diffusion bridge network, the encoder in the perception image compression network mapping the remote sensing sample image and the corresponding transformed sample image from the pixel space to the feature space, the denoising diffusion bridge network performing forward diffusion and reverse denoising in the feature space, and the decoder in the perception image compression network mapping the output of the denoising diffusion bridge network from the feature space to the pixel space;

[0009] Obtaining a real-time remote sensing image, inputting the real-time remote sensing image into a trained online map generation model, and obtaining a real-time network map.

[0010] In one embodiment, in training the online map generation model, the remote sensing sample image is taken as the end point of forward diffusion and the starting point of reverse denoising in the denoising diffusion bridge network.

[0011] In one embodiment, in training the online map generation model:

[0012] The remote sensing sample image and the corresponding multiple transformed sample images are respectively mapped to a feature space by an encoder in the perception image compression network, to obtain a remote sensing sample feature map and multiple transformed sample feature maps;

[0013] The denoising diffusion bridge network is used to obtain a predicted network map in the feature space and multiple predicted transformed network maps under the guidance of the corresponding network map according to the remote sensing sample feature map and the multiple transformed sample feature maps respectively;

[0014] The predicted network map in the feature space and the multiple predicted transformed network maps are respectively mapped to a pixel space by a decoder in the perception image compression network, to obtain a predicted network map in the pixel space and multiple predicted transformed network maps in the pixel space;

[0015] The predicted network map in the pixel space is geometrically transformed to obtain a transformed predicted network map consistent with the transformation mode of each predicted transformed network map;

[0016] A calculation transformation consistency regularization loss function is calculated according to the multiple predicted transformed network maps and the multiple transformed predicted network maps, and parameters in the online map generation model are adjusted according to the calculation result until convergence, to obtain a trained online map generation model.

[0017] In one embodiment, in training the online map generation model;

[0018] The perception image compression network and the denoising diffusion bridge network each include two units for processing the remote sensing sample image and the transformed sample image;

[0019] The perception image compression network includes a first compression unit for processing the remote sensing sample image and a second compression unit for processing the multiple transformed sample images;

[0020] The denoising diffusion bridge network includes a first diffusion unit for processing the remote sensing sample image and a second diffusion unit for processing the multiple transformed sample images, and the parameters of the first diffusion unit and the second diffusion unit are shared.

[0021] In an embodiment, the trained online map generation model is composed of a first compression unit and a first diffusion unit.

[0022] In an embodiment, when multiple different transformed sample images are obtained by geometric transformation on the remote sensing sample image, rotation, clipping and flipping transformation are respectively performed on the remote sensing image.

[0023] In an embodiment, the perception image compression network in the online map generation model adopts a pre-trained vector quantization generative adversarial network.

[0024] The application also provides a device for generating an online map based on a diffusion model of a remote sensing image, the device comprising:

[0025] A sample data set acquisition module is configured to acquire a sample image data set, wherein the sample image data set comprises multiple groups of sample image pairs, and each group of sample image pairs comprises a remote sensing sample image and a corresponding network map.

[0026] A training data set construction module is configured to obtain multiple different transformed sample images by performing geometric transformation on the remote sensing sample image in each group of sample image pairs, add the multiple transformed sample images to the corresponding sample image pair to obtain a training data group, and construct a training data set according to multiple training data groups.

[0027] A model training module is configured to train an online map generation model using the training data set to obtain a trained online map generation model, wherein the online map generation model comprises a perception image compression network and a denoising diffusion bridge network, the encoder in the perception image compression network is configured to map the remote sensing sample image and the corresponding transformed sample image from a pixel space to a feature space, the denoising diffusion bridge network is configured to perform forward diffusion and reverse denoising in the feature space, and the decoder in the perception image compression network is configured to map the output of the denoising diffusion bridge network from the feature space to the pixel space.

[0028] An online map generation module is configured to acquire a real-time remote sensing image, input the real-time remote sensing image into the trained online map generation model, and obtain a real-time network map.

[0029] A computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0030] A sample image data set is acquired, wherein the sample image data set comprises multiple groups of sample image pairs, and each group of sample image pairs comprises a remote sensing sample image and a corresponding network map.

[0031] geometric transformation is performed on the remote sensing sample image in each group of sample image pairs to obtain a plurality of different transformed sample images, the plurality of transformed sample images are added to the corresponding sample image pairs to obtain a training data group, and a training data set is constructed according to a plurality of training data groups;

[0032] The training data set is used to train an online map generation model to obtain a trained online map generation model, the online map generation model includes a perception image compression network and a denoising diffusion bridge network, an encoder in the perception image compression network is used to map the remote sensing sample image and the corresponding transformed sample image from a pixel space to a feature space, the denoising diffusion bridge network is used to perform forward diffusion and reverse denoising in the feature space, and an output of the denoising diffusion bridge network is mapped from the feature space to the pixel space by a decoder in the perception image compression network;

[0033] A real-time remote sensing image is obtained, and the real-time remote sensing image is input into the trained online map generation model to obtain a real-time network map.

[0034] A computer readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to implement the following steps:

[0035] A sample image data set is obtained, and the sample image data set includes a plurality of groups of sample image pairs, each group of sample image pairs includes a remote sensing sample image and a corresponding network map;

[0036] Geometric transformation is performed on the remote sensing sample image in each group of sample image pairs to obtain a plurality of different transformed sample images, the plurality of transformed sample images are added to the corresponding sample image pairs to obtain a training data group, and a training data set is constructed according to a plurality of training data groups;

[0037] The training data set is used to train an online map generation model to obtain a trained online map generation model, the online map generation model includes a perception image compression network and a denoising diffusion bridge network, an encoder in the perception image compression network is used to map the remote sensing sample image and the corresponding transformed sample image from a pixel space to a feature space, the denoising diffusion bridge network is used to perform forward diffusion and reverse denoising in the feature space, and an output of the denoising diffusion bridge network is mapped from the feature space to the pixel space by a decoder in the perception image compression network;

[0038] A real-time remote sensing image is obtained, and the real-time remote sensing image is input into the trained online map generation model to obtain a real-time network map.

[0039] The remote sensing image generation online map method, device, equipment and medium, through geometric transformation on the remote sensing sample images in each group of sample image pairs to obtain a plurality of different transformed sample images, adding the plurality of transformed sample images to the corresponding sample image pairs to obtain a training data group, constructing a training data set according to a plurality of training data groups, training the online map generation model by using the training data set to obtain a trained online map generation model, mapping the remote sensing sample images and the corresponding transformed sample images from a pixel space to a feature space by using an encoder in the perceptual image compression network, performing forward diffusion and reverse denoising in the feature space by using a denoising diffusion bridge network, and mapping the output of the denoising diffusion bridge network from the feature space to the pixel space by using a decoder in the perceptual image compression network, inputting the real-time remote sensing image into the trained online map generation model to obtain a real-time network map. The method can generate a map with clearer boundaries and brighter colors in real time. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 A schematic diagram of local details inaccurate and semantic distortion for an existing RSMT method in an embodiment;

[0041] Figure 2 A flowchart of a remote sensing image generation online map method in an embodiment;

[0042] Figure 3 A structure diagram of a perceptual image compression network in an embodiment;

[0043] Figure 4 A structure diagram of an online map generation model in an embodiment;

[0044] Figure 5 A qualitative result based on a first data set in an experiment;

[0045] Figure 6 A qualitative result based on a second data set in an experiment;

[0046] Figure 7 An ablation experiment result in an experiment;

[0047] Figure 8 A structure block diagram of a remote sensing image generation online map device in an embodiment;

[0048] Figure 9 An internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0050] In the prior art, there are certain limitations in using a generative adversarial network in a domain mapping method to convert remote sensing images into network maps. It still lacks in capturing more diversity, and the lack of pattern coverage may lead to blurred boundaries and semantic distortion in the conversion of remote sensing images into maps, as shown in FIG. 1.

[0051] In view of the above problems, as shown in FIG. 1, a remote sensing image generation online map method based on a diffusion model is provided, and the specific steps include: Figure 2

[0052] Step S100, a sample image data set is obtained, and the sample image data set includes a plurality of groups of sample image pairs. Each group of sample image pairs includes a remote sensing sample image and a corresponding network map.

[0053] Step S110, geometric transformation is performed on the remote sensing sample image in each group of sample image pairs to obtain a plurality of different transformed sample images. The plurality of transformed sample images are added to the corresponding sample image pairs to obtain a plurality of training data groups. The training data set is constructed according to the plurality of training data groups.

[0054] Step S120, the training data set is used to train an online map generation model to obtain a trained online map generation model. The online map generation model includes a perception image compression network and a denoising diffusion bridge network. The encoder in the perception image compression network maps the remote sensing sample image and the corresponding transformed sample image from the pixel space to the feature space. The denoising diffusion bridge network performs forward diffusion and reverse denoising in the feature space. The output of the denoising diffusion bridge network is mapped from the feature space to the pixel space by the decoder in the perception image compression network.

[0055] Step S130, a real-time remote sensing image is obtained, and the real-time remote sensing image is input into the trained online map generation model to obtain a real-time network map.

[0056] In the present embodiment, a new end-to-end method based on a denoising diffusion model is proposed for generating an online network map from a remote sensing image. In the denoising diffusion model, a Brownian motion strategy is used to balance diversity and accuracy. Further, in order to reduce the randomness of the conditional guided down-sampling process, a denoising diffusion bridge model is used in the present method to directly learn the distribution between the remote sensing image and the online network map through a bidirectional diffusion process.

[0057] ​In the embodiment, the above steps are divided into two parts, steps S100 to S120 are the part of model training, and step S130 is the part of applying the trained model.

[0058] In step S100, the sample image dataset includes multiple groups of sample image pairs, each group of sample image pairs includes a remote sensing image and a corresponding network map. The network map refers to an online map tile based on the Internet, that is, a kind of digital map. The network map is used as the true value label when training the online map generation model.

[0059] In step S110, in order to increase the diversity of training data, multiple different geometric transformations are performed on the remote sensing image to obtain multiple transformed sample images, and the generated multiple transformed sample images are added to the corresponding sample image pair to obtain a training data group. This operation is performed on the remote sensing image in each sample image pair to obtain multiple training data groups, and then a training dataset is obtained. Subsequently, when training the online map generation model, each training data group is sequentially input into the model for training.

[0060] In the embodiment, the remote sensing image is geometrically transformed, and the transformed image is used as the training image, which can increase the diversity of training data and enable the model to achieve good results under a large amount of training data. The model robustness can also be improved. By performing geometric transformation on the training data, the model can learn more feature transformation conditions, thereby having better robustness to different input images. In this way, the model can more stably generate a map when facing various changes in real scenes. At the same time, overfitting can be reduced, the generalization ability of the model can be improved, real-world changes can be simulated, and the recognition ability of spatial transformation can be enhanced.

[0061] Specifically, when the remote sensing sample image is geometrically transformed to obtain multiple different transformed sample images, the remote sensing image is respectively rotated, cropped, and flipped. That is, each remote sensing sample image is geometrically transformed to obtain a corresponding rotated transformed sample image, a cropped transformed sample image, and a flipped transformed sample image.

[0062] In step S120, the online map generation model includes a perception image compression network and a denoising diffusion bridge network. The perception image compression network is used to project the sample image from the pixel domain to the feature space, the denoising diffusion bridge network is used to perform forward diffusion and reverse denoising in the feature space, and then the perception image compression network is used to project the result back to the pixel space.

[0063] In the embodiment, in order to improve the training effect and the result quality, the encoder and the decoder of the pre-trained perception image compression network are used to map the diffusion process from the pixel space to the latent feature space.

[0064] Specifically, after inputting a training data set into the online map generation model, the encoder in the perception image compression network is used to respectively map the remote sensing sample image and the corresponding multiple transformed sample images to the feature space, to obtain the remote sensing sample feature map and the multiple transformed sample feature maps. The denoising diffusion bridge network is used to obtain the predicted network map in the feature space and the multiple predicted transformed network maps according to the remote sensing sample feature map and the multiple transformed sample feature maps under the guidance of the corresponding network map, and the decoder in the perception image compression network is used to respectively map the predicted network map in the feature space and the multiple predicted transformed network maps to the pixel space, to obtain the predicted network map in the pixel space and the multiple predicted transformed network maps. The predicted network map in the pixel space is geometrically transformed to obtain the transformed predicted network map consistent with the transformation mode of each predicted transformed network map, the multiple predicted transformed network maps and the multiple transformed predicted network maps are used to calculate the transformation consistency regularization loss function, the parameters in the online map generation model are adjusted according to the calculation result until convergence, and the trained online map generation model is obtained.

[0065] Further, when training the model, the denoising diffusion bridge network includes two diffusion processes: and . Wherein, the subscripts A and B respectively represent the remote sensing image domain and the network map domain, and represent the remote sensing sample image and the corresponding network map in a training data set, and the superscript tilde represents the transformed sample image.

[0066] First, from the domains and , the is obtained, and the input is mapped to the latent space. The feature vector spaces and are the two endpoints of the diffusion bridge. Second, inspired by consistency regularization, the transformation consistency regularization (TCR) is used to maintain the geometric accuracy of the results. is a geometric transformation function, which aims to explain the regularization transformation. and are also mapped to the latent space and after geometric transformation. Finally, the total loss function of the trained model contains two parts: one is the evidence lower bound (ELBO) for the denoising diffusion bridge model; the other is the TCR loss for the entire online map generation model, which can be regarded as a reconstruction loss, which depends on the predefined geometric transformation function .

[0067] In view of the characteristics of the remote sensing image, in the training process of the model, the consistency regularization loss function is also changed to constrain it in the embodiment. In the actual scene, there are still differences between the real-time remote sensing image obtained and the training sample image, so in the training process, the change consistency regularization needs to be used to ensure the generalization of the model.

[0068] Considering that the traditional diffusion model is limited by its huge computing source cost. Therefore, in the embodiment, a pre-trained VQGAN (Vector Quantized Generative Adversarial Network) is used to realize perceptual image compression (Perceptual Image Compression), and the specific architecture is as shown in Figure 3 .

[0069] Specifically, the pre-trained VQGAN includes three parts: an encoder, a decoder, and image regularization. Given an image in the domain , the encoder maps to a latent representation , where , and the decoder reconstructs the image from the latent space . In particular, the encoder down-samples the image by the coefficient , where .

[0070] At the same time, in order to avoid an arbitrarily high-variance latent space, two regularization methods, namely KL-reg and VQ-reg, are proposed. The KL-reg applies a slight standard normal KL-penalty to the learned latent space. The VQ-reg is a vector quantization layer in the decoder, which first reshapes the feature vector. Then, by comparing the distance between it and the vectors in the embedding dictionary, the appropriate feature vector with the smallest distance in the dictionary can be selected.

[0071] When training the online map generation model, in the forward diffusion process of the denoising diffusion bridge network, unlike the existing diffusion model with pure Gaussian noise as the diffusion endpoint, in the embodiment, the remote sensing sample image is taken as the end point of the forward diffusion in the denoising diffusion bridge network, that is, the starting point of the reverse denoising.

[0072] In this way, the sample data . If the conditional probability is obtained by evolution, the time-reversal SDE is composed of formula 1, which is expressed as:

[0073] (1)

[0074] In formula (1), Therefore, given an endpoint remote sensing sample image , the corresponding mapping is recovered according to the SDE equation.

[0075] However, the general denoising diffusion bridge model is constructed based on various random processes, which may cause randomness in the output quality. Therefore, in this embodiment, the strategy of Brownian motion is adopted, which makes the mathematical process easier to understand. At each time in the diffusion process, given and the end point , the process can be expressed as:

[0076] (2)

[0077] (3)

[0078] In formula (2), is the total number of diffusion processes, is the variance.

[0079] If the distribution should be a standard normal distribution, then the variance of the intermediate step should remain consistent. However, according to formula (2), as increases, will become very large. Therefore, in order to obtain a relatively controllable variance, a new Brownian bridge diffusion process variance table can be designed, and its calculation formula is:

[0080] (4)

[0081] As can be seen from formula (4), the variance reaches a maximum value at the middle time . Then the variance decreases until . Therefore, an influence factor is also introduced, which can scale to control the diversity of the sampling results.

[0082] In addition to the marginal distribution, the forward transition probability also needs to be derived. According to formula (3), only the marginal distribution can be obtained at each step. In the forward process, the forward transition probability needs to be derived. By calculating at each step, the transition probability can be obtained:

[0083] (5)

[0084] When the diffusion process reaches the destination , the following can be obtained​ and . In this way, the translation between A and B domains is determined.

[0085] Further, in the reverse denoising process, in the existing diffusion model, the starting point of the reverse process is the pure noise sampled from the Gaussian distribution. The purpose of the diffusion process is to gradually eliminate the noise to obtain the objective result. However, for a specific RSMT task, there are still many uncontrollable diversities in this conditional transformation. The proposed diffusion bridge process starts from the conditional reference, that is, . The main calculation of prediction is as follows.

[0086] (6)

[0087] In formula (6), is the mean value of the prediction, which needs to learn the parameters trained from the neural network. The variance plays an important role in the high-quality result and is related to the objective function. In the reverse process, the reference value sampled from the remote sensing image domain starts.

[0088] In this embodiment, the total loss function of the online map generation model can be represented as:

[0089] (7)

[0090] In formula (7), is the lower bound of the evidence, is the variation consistency loss, and are the settable calculation weights.

[0091] Specifically, in the training process of the denoising diffusion bridge model, the lower bound of the evidence (ELBO) of the Brownian bridge diffusion process is optimized, which can be expressed as:

[0092] (8)

[0093] Since, in this embodiment, the diffusion endpoint is fixed, that is, , the first term in formula (8) can be regarded as a constant. Combining the above formula (3) and formula (8), the formula in the second term can be obtained by Bayes theorem and Markov chain characteristics:

[0094] (9)

[0095] In formula (9), the key is to calculate . At the same time, the neural network Predictive noise, rather than the entire So The expression is:

[0096] (10)

[0097] In formula (10), , and The calculation details are omitted in the article.

[0098] Therefore, according to the above analysis, the objective function of the denoising diffusion bridge model training can be expressed as:

[0099] (11)

[0100] In formula (11), Indicates , And the average value of noise The above design of a factor. To be regarded as random noise, to increase randomness. Is a noise function that changes with time step The entire formula shows that based on the reference value , the original input value And the noise value , the modal can be optimized to make the generated sample closer to the target distribution.

[0101] Specifically, consistency regularization (CR) has shown advanced performance in classification tasks. When the input sample is disturbed, consistency regularization will force the model to maintain consistent prediction results when facing various data disturbances. Under the incentive of CR, transformation consistency regularization (TCR) is applied to RMST. Through geometric transformation, the model is forced to generate a reconstructed map consistent with the original Figure 1 .

[0102] Further, when calculating the consistency regularization loss function, the predicted network map obtained by predicting the remote sensing sample image will be geometrically transformed to obtain multiple transformed predicted network maps, which will be respectively calculated with the predicted transformation network map using the same geometric transformation method. Consistency regularization loss function.

[0103] In this embodiment, when the remote sensing sample image is geometrically transformed, rotation, clipping and flipping are used, and correspondingly, the same geometric transformation is performed on the predicted network map after it is obtained.

[0104] Specifically, the TCR loss function is represented as:​

[0105] (12)

[0106] In formula (12), the geometric transformation operation is performed on the remote sensing sample image , i.e., the sample image is transformed, the prediction is performed on the transformed sample image using the diffusion model to obtain a transformed prediction network map, and the prediction is performed on the remote sensing sample image using the diffusion model and then the geometric transformation is performed. The loss function ensures the consistency between the maps reconstructed from the given reference remote sensing image.

[0107] In the present embodiment, when the online map generation model is trained: the perception image compression network and the denoising diffusion bridge network each include two units for processing the remote sensing sample image and the transformed sample image, the perception image compression network includes a first compression unit for processing the remote sensing sample image and a second compression unit for processing a plurality of the transformed sample images, the denoising diffusion bridge network includes a first diffusion unit for processing the remote sensing sample image and a second diffusion unit for processing a plurality of the transformed sample images, and the parameters of the first diffusion unit and the second diffusion unit are shared.

[0108] In one embodiment, the structure of the online map generation model and the diffusion process of the denoising diffusion bridge model are as shown in Figure 4 .

[0109] In step S130, only the first compression unit and the first diffusion unit are included in the trained online map generation model. The real-time remote sensing image is first mapped to the feature space by the first compression unit, and then the real-time remote sensing image mapped to the feature space is used to generate the corresponding network map feature by the first diffusion unit. The network map feature is restored to the pixel space by the first compression unit to obtain the final real-time network map.

[0110] In this paper, the effectiveness of the method is also verified by experiments.

[0111] In order to verify the effectiveness of the proposed model, experiments were conducted on two different data sets corresponding to different regions.

[0112] The first data set: 4631 pairs of remote sensing images and maps from a certain region were selected. The zoom level was 17, and the spatial resolution was 2.15 meters / pixel.

[0113] The second data set: remote sensing images and maps of another region were selected, the zoom level was 17, and the spatial resolution was 2.15 meters / pixel. In order to remove "dirty data", 3200 accurately matched paired samples were selected.

[0114] ​​​After analyzing the two datasets, it was found that the coloring of the same geographical features was inconsistent, and the paired samples were not accurate and complete. After removing the mismatched samples, fully paired samples were obtained to support the accurate generation of online maps.

[0115] To evaluate the performance of the proposed method (MapGen-Diff), the translation results were compared with several baseline methods through quantitative and qualitative indicators. In the experiment, some SOTA image-to-image translation methods, Pix2pix, CycleGAN, and AIME, and several RSMT methods, MapGenGAN, SMAPGAN, and SemiMap-Gen, were selected. Typical translation models were selected to evaluate the performance of the proposed method. For supervised learning models, fully paired samples in the above datasets were used. Then, all paired remote sensing images and map samples in each dataset were shuffled to become unpaired samples for CycleGAN and MapGenGAN. The experiment on SMAPGAN and Semi-MapGen made the ratio of paired samples and unpaired samples 1:4.

[0116] To demonstrate the effectiveness of the proposed method in generating map tiles from remote sensing images, the proposed MapGen-Diff method was compared with six typical models of RMST.

[0117] As shown in Figure 5 and Figure 6 , the qualitative results obtained by conducting experiments on two datasets, respectively. The experiment selected a part of the samples generated from the maps of the two different datasets for demonstration. In the comparison chart, each column corresponds to the input remote sensing image, ground truth, map generated by the proposed method, and map generated by other baseline models, respectively.

[0118] Overall, the proposed method outperforms other comparison models in identifying vegetation and waterway colors and identifying road and building boundaries. This performance improvement can be attributed to the geometric constraints and diversity provided by the diffusion model, which helps the proposed method to obtain visually striking results. The addition of these elements makes MapGen-Diff stand out from its counterparts in RSMT in terms of outstanding visual performance.

[0119] As can be seen from Figure 5 , the maps generated by the unsupervised methods CycleGAN and MapGen-GAN are somewhat blurred. Semi-supervised methods are effective in reconstructing the shape of roads and buildings, but cannot correctly identify the differences in feature elements. In terms of overall visual quality, Pix2pix and ATME are significantly inferior to the proposed method.

[0120] In the second dataset, which includes multiple geographic elements such as streets, highways, vegetation, and waterways, it is more challenging to generate corresponding images with clear boundaries and shapes. From Figure 6 As can be seen from Table 1, MapGen-Diff can explicitly distinguish different ground elements and has diversity and accuracy. The supervised methods Pix2pix and ATME generate some semantic distorted maps and are slightly inferior to our method in terms of color representation of different ground elements. Unsupervised and semi-supervised methods show limitations in the accuracy of different area classification. In addition, for subtle differences between remote sensing images and ground truth, our method can learn to represent colors close to remote sensing images.

[0121] Further, to evaluate the objective indicators of our method compared with other baseline methods, three image quality evaluation indicators are adopted. Tables 1 and 2 list the quantitative results of each method on the first and second datasets, respectively. The highest score is in bold and the runner-up is underlined. Our method is compared with supervised, unsupervised, and semi-supervised methods, respectively.

[0122] Overall, our method outperforms other baseline methods in almost all indicators, while showing sub-optimal performance in ACC on the first dataset. It is acknowledged that ACC involves pixel-level comparison of RGB values between the original image and the translated image. Therefore, subtle differences in RGB values will also affect the ACC score. Our method is based on a diffusion model, which is known for its diverse distribution. Therefore, sub-optimal performance can be accepted without affecting the quality of generation.

[0123] Specifically, first, for the first dataset, our method model achieves the highest score in both RMSE and SSIM indicators. Although our method slightly lags behind ATME in the ACC indicator, it still achieves a commendable score of 61%. In particular, compared with previous methods, the RMSE of MapGen-Diff is improved by 5%-30% and the SSIM is improved by 2%-17%.

[0124] Second, in the second dataset, for the translation task from remote sensing images to maps, the proposed framework of our method is significantly superior to all baseline methods in the three performance indicators. Compared with the average performance of other baselines, it is significantly improved by 29% in RMSE, 8% in SSIM, and 14% in ACC. Compared with the first dataset, our method shows higher stability on the second dataset. This improvement in stability indicates that the diffusion model has higher robustness and reliability, which can be attributed to the change in data complexity and geographical features represented by the second dataset.

[0125] Table 1 Quantitative results on the first dataset

[0126]

[0127] Table 2 Quantitative results on the second dataset

[0128]

[0129] Further, an ablation study is conducted to verify the effectiveness of each component in the online map generation model and the influence of several parameters, the experimental results are shown in Figure 7 Figure 6, where the first column MapGen-Diff-no-L TCR represents the removal of the LCR loss function, the red boxes highlight the places where the effect is worse on the qualitative results, the second column MapGen-Diff-no-Latent represents the removal of the operation of mapping to the feature space, the resulting image becomes blurred, and the last three columns represent the influence of the maximum variance in the diffusion process on the results, which is the influence of the parameters in the diffusion process.

[0130] In the above-mentioned online map generation method based on the diffusion model, a new online map conversion model based on the diffusion model is proposed. In terms of map generation, this method can widely realize high-fidelity generation function based on the denoising diffusion bridge model. Inspired by the Brownian motion strategy, the general diffusion model is improved, and a trade-off between the diversity and accuracy of the generated map is made. By mapping the image to the latent space, the network can capture more features of the ground elements. With the help of the transformation regularization consistency loss, this method can generate maps with clearer boundaries and more vibrant colors. After extensive experimental comparison on two widely used datasets, it is proved that this method is more competitive than the previous map generated by the remote sensing image to map method in both objective and subjective aspects.

[0131] It should be understood that, although Figure 1 the steps in the flowchart of Figure 1 may appear to be in a particular order, these steps are not necessarily performed in the order shown by the arrows. Unless specifically stated otherwise in this document, the steps can be performed in other orders. Moreover, at least some of the steps in

[0132] may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but can be performed at different times, and the order of the sub-steps or stages is not necessarily sequential, but can be performed alternately or alternately with at least some of the other steps or sub-steps or stages of other steps. Figure 8 In one embodiment, asAs shown, a diffusion model-based remote sensing image online map generation device is provided, comprising: a sample data set acquisition module 200, a training data set construction module 210, a model training module 220 and an online map generation module 230, wherein:

[0133] The sample data set acquisition module 200 is configured to acquire a sample image data set, wherein the sample image data set comprises a plurality of groups of sample image pairs, each group of sample image pairs comprises a remote sensing sample image and a corresponding network map.

[0134] The training data set construction module 210 is configured to perform geometric transformation on the remote sensing sample image in each group of sample image pairs to obtain a plurality of different transformed sample images, add the plurality of transformed sample images to the corresponding sample image pair to obtain a training data group, and construct a training data set according to a plurality of training data groups.

[0135] The model training module 220 is configured to train an online map generation model using the training data set to obtain a trained online map generation model, wherein the online map generation model comprises a perception image compression network and a denoising diffusion bridge network, the encoder in the perception image compression network maps the remote sensing sample image and the corresponding transformed sample image from a pixel space to a feature space, the denoising diffusion bridge network performs forward diffusion and reverse denoising in the feature space, and the decoder in the perception image compression network maps the output of the denoising diffusion bridge network from the feature space to the pixel space.

[0136] The online map generation module 230 is configured to acquire a real-time remote sensing image, input the real-time remote sensing image into the trained online map generation model, and obtain a real-time network map.

[0137] The specific limitations of the diffusion model-based remote sensing image online map generation device can be referred to the limitations of the diffusion model-based remote sensing image online map generation method described above, which will not be repeated here. Each module in the above diffusion model-based remote sensing image online map generation device can be realized by software, hardware and their combination in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to call and execute the operations corresponding to each module by the processor.

[0138] In one embodiment, a computer device is provided, which can be a terminal, and its internal structure diagram can be as shown in Figure 9As shown in the figure. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with the external terminal through the network connection. The computer program is executed by the processor to implement a method for generating an online map based on a diffusion model from a remote sensing image. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0139] Those skilled in the art can understand that, Figure 9 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. A specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0140] In one embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the following steps:

[0141] Obtain a sample image data set, which includes a plurality of groups of sample image pairs, each group of sample image pairs including a remote sensing sample image and a corresponding network map;

[0142] Geometrically transform the remote sensing sample image in each group of sample image pairs to obtain a plurality of different transformed sample images, add the plurality of transformed sample images to the corresponding sample image pair to obtain a training data group, and construct a training data set according to a plurality of training data groups;

[0143] Train the online map generation model using the training data set to obtain a trained online map generation model, the online map generation model including a perception image compression network and a denoising diffusion bridge network, the encoder in the perception image compression network mapping the remote sensing sample image and the corresponding transformed sample image from the pixel space to the feature space, the denoising diffusion bridge network performing forward diffusion and reverse denoising in the feature space, and the decoder in the perception image compression network mapping the output of the denoising diffusion bridge network from the feature space to the pixel space;

[0144] Obtaining a real-time remote sensing image, inputting the real-time remote sensing image into the trained online map generation model to obtain a real-time network map.

[0145] In one embodiment, a computer readable storage medium is provided, and a computer program is stored on the computer readable storage medium, and the computer program is executed by a processor to implement the following steps:

[0146] Obtaining a sample image dataset, the sample image dataset including a plurality of groups of sample image pairs, each group of sample image pairs including a remote sensing sample image and a corresponding network map;

[0147] Performing geometric transformation on the remote sensing sample image in each group of sample image pairs to obtain a plurality of different transformed sample images, adding the plurality of transformed sample images to the corresponding sample image pair to obtain a training data group, and constructing a training dataset according to a plurality of training data groups;

[0148] Training the online map generation model using the training dataset to obtain a trained online map generation model, the online map generation model including a perception image compression network and a denoising diffusion bridge network, the encoder in the perception image compression network mapping the remote sensing sample image and the corresponding transformed sample image from a pixel space to a feature space, the denoising diffusion bridge network performing forward diffusion and reverse denoising in the feature space, and the decoder in the perception image compression network mapping the output of the denoising diffusion bridge network from the feature space to the pixel space;

[0149] Obtaining a real-time remote sensing image, inputting the real-time remote sensing image into the trained online map generation model to obtain a real-time network map.

[0150] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, databases, or other media in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0151] The technical features of the above embodiments can be combined in any way. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, but as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0152] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of the patent protection of the present application should be subject to the appended claims.

Claims

1. A method for generating online maps from remote sensing images based on a diffusion model, characterized in that, The method includes: Obtain a sample image dataset, which includes multiple pairs of sample images, each pair of sample images including a remote sensing sample image and a corresponding network map; Geometric transformations are performed on the remote sensing sample images in each set of sample image pairs to obtain multiple different transformed sample images. These multiple transformed sample images are added to the corresponding sample image pairs to obtain training data sets. A training dataset is constructed based on the multiple training data sets. The online map generation model is trained using the training dataset to obtain a trained online map generation model. The online map generation model includes a perceptual image compression network and a denoising diffusion bridge network. The encoder in the perceptual image compression network maps the remote sensing sample image and the corresponding transformed sample image from the pixel space to the feature space. The denoising diffusion bridge network performs forward diffusion and reverse denoising in the feature space. Then, the decoder in the perceptual image compression network maps the output of the denoising diffusion bridge network from the feature space to the pixel space. Acquire real-time remote sensing images and input them into a trained online map generation model to obtain a real-time network map.

2. The method for generating online maps from remote sensing images according to claim 1, characterized in that, When training the online map generation model, the remote sensing sample image is used as the endpoint of forward diffusion in the denoising diffusion bridge network, which is the starting point of reverse denoising.

3. The method for generating online maps from remote sensing images according to claim 2, characterized in that, When training the online map generation model: The encoder in the sensor image compression network maps the remote sensing sample image and the corresponding multiple transformed sample images to the feature space to obtain the remote sensing sample feature map and the multiple transformed sample feature maps. Using the denoising diffusion bridge network, based on the remote sensing sample feature map and multiple transformed sample feature maps, and guided by the corresponding network map, a predicted network map of the feature space and multiple predicted transformed network maps are obtained. The decoder in the perceptual image compression network then maps the prediction network map of the feature space and multiple prediction transformation network maps to the pixel space to obtain the prediction network map and multiple prediction transformation network maps of the pixel space. Geometric transformation is performed on the prediction network map in the pixel space to obtain a transformed prediction network map that is consistent with the transformation method of each prediction transformation network map. The transformation consistency regularization loss function is calculated based on multiple predicted transformation network maps and multiple transformed prediction network maps. The parameters in the online map generation model are adjusted according to the calculation results until convergence, thus obtaining the trained online map generation model.

4. The method for generating online maps from remote sensing images according to claim 3, characterized in that, When training the online map generation model; Both the sensing image compression network and the denoising diffusion bridge network include two units: one for processing the remote sensing sample image and the other for transforming the sample image. The sensing image compression network includes a first compression unit that processes the remote sensing sample images, and a second compression unit that processes multiple transformed sample images. The denoising diffusion bridge network includes a first diffusion unit that processes the remote sensing sample feature map and a second diffusion unit that processes multiple transformed sample feature maps, wherein the parameters of the first diffusion unit and the second diffusion unit are shared.

5. The method for generating online maps from remote sensing images according to claim 4, characterized in that, The trained online map generation model consists of a first compression unit and a first diffusion unit.

6. The method for generating online maps from remote sensing images according to any one of claims 1-5, characterized in that, When performing geometric transformations on the remote sensing sample images to obtain multiple different transformed sample images, the remote sensing images are rotated, cropped, and flipped respectively.

7. The method for generating online maps from remote sensing images according to any one of claims 1-5, characterized in that, The perceptual image compression network in the online map generation model employs a pre-trained vector quantization generative adversarial network.

8. An online map generation device based on a diffusion model for remote sensing images, characterized in that, The device includes: The sample dataset acquisition module is used to acquire a sample image dataset, which includes multiple sets of sample image pairs, and each set of sample image pairs includes a remote sensing sample image and a corresponding network map. The training dataset construction module is used to perform geometric transformations on the remote sensing sample images in each set of sample image pairs to obtain multiple different transformed sample images, add the multiple transformed sample images to the corresponding sample image pairs to obtain training data sets, and construct a training dataset based on the multiple training data sets. The model training module is used to train the online map generation model using the training dataset to obtain a trained online map generation model. The online map generation model includes a perceptual image compression network and a denoising diffusion bridge network. The encoder in the perceptual image compression network maps the remote sensing sample image and the corresponding transformed sample image from the pixel space to the feature space. The denoising diffusion bridge network performs forward diffusion and reverse denoising in the feature space. Then, the decoder in the perceptual image compression network maps the output of the denoising diffusion bridge network from the feature space to the pixel space. The online map generation module is used to acquire real-time remote sensing images and input the real-time remote sensing images into a trained online map generation model to obtain a real-time network map.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images