Multi-modal sample generation method based on geographic information conditional diffusion model

Abstract

The invention provides a multi-modal sample generation method based on a geographic information conditional diffusion model, and relates to the technical field of remote sensing image generation, and the method comprises the steps: obtaining a remote sensing multi-modal semantic segmentation data set which comprises a multi-source image and a semantic tag corresponding to the multi-source image; inputting the remote sensing multi-modal semantic segmentation data set into a geographic information condition diffusion model for training to obtain a plurality of multi-modal samples; wherein the geographic information condition diffusion model comprises a joint input module and a data embedding module; performing channel splicing processing on a multi-source image and a semantic tag in the remote sensing multi-mode semantic segmentation data set through a joint input module to obtain target image tag data; the method comprises the following steps: performing a noise injection-noise reduction process and data decomposition processing on target image label data through a data embedding module to obtain a plurality of multi-modal samples; according to the method, the problem that a remote sensing image annotation data sample generally lacks text description information is solved, and low-cost and high-efficiency data sample generation is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image generation, and in particular to a multimodal sample generation method, device, electronic device and storage medium based on a geographic information conditional diffusion model. Background Art

[0002] As an important part of the earth's resources, mines are vital to the development of social economy. However, mining activities may be accompanied by environmental changes and ecosystem damage, so monitoring and managing land cover changes in mining areas is crucial for ecological, environmental and social development. Remote sensing images are one of the important data resources for recording surface information. Through deep learning methods, remote sensing images can be automatically interpreted and analyzed, which is suitable for applications such as scene classification, semantic segmentation and instance segmentation, providing an effective tool for mine monitoring and management.

[0003] However, building a pre-trained model for remote sensing images requires a large amount of annotated datasets, and annotating remote sensing images is an expensive and time-consuming task. Using traditional image enhancement techniques may not be able to fully simulate the sample diversity of remote sensing images. Therefore, a common solution is to use data enhancement to increase sample diversity and make full use of existing labels. However, traditional image enhancement techniques (such as flipping, rotation, and scaling) are usually not enough to simulate the sample diversity of satellite images. Summary of the invention

[0004] The problem solved by the present invention is how to solve the problem that remote sensing image annotation data samples usually lack text description information.

[0005] In order to solve the above problems, the present invention provides a multimodal sample generation method, device, electronic device and storage medium based on a geographic information conditional diffusion model.

[0006] In a first aspect, the present invention provides a multimodal sample generation method based on a geographic information conditional diffusion model, comprising:

[0007] Acquire a remote sensing multimodal semantic segmentation dataset, wherein the remote sensing multimodal semantic segmentation data includes multi-source images and semantic labels corresponding to the multi-source images;

[0008] Inputting the remote sensing multimodal semantic segmentation data set into a geographic information conditional diffusion model for training to obtain a plurality of multimodal samples;

[0009] Wherein, the geographic information conditional diffusion model includes a joint input module and a data mosaic module;

[0010] The multi-source images and semantic labels in the remote sensing multimodal semantic segmentation dataset are subjected to channel splicing processing through the joint input module to obtain target image label data;

[0011] The target image label data is subjected to a noise injection-noise reduction process and a data decomposition process through the data embedding module to obtain a plurality of the multimodal samples.

[0012] Optionally, the acquiring of a remote sensing multimodal semantic segmentation dataset includes:

[0013] Obtain the image of the current study area and the mine classification and annotation data of the current study area;

[0014] According to the current study area image and the mine sub-classification annotation data, a plurality of the multi-source images and a plurality of the semantic labels are obtained;

[0015] The remote sensing multimodal semantic segmentation dataset is obtained according to the multiple multi-source images and the multiple semantic labels.

[0016] Optionally, the step of performing channel stitching processing on multi-source images and semantic labels in the remote sensing multimodal semantic segmentation dataset through the joint input module to obtain target image label data includes:

[0017] Performing standardization processing on the multi-source images to obtain a standardized image;

[0018] Mapping the discrete label value corresponding to the semantic label into a binary vector representation to obtain a binary label;

[0019] The standardized image and the binary label are concatenated in the channel dimension to obtain target image label data.

[0020] Optionally, the data mosaic module performs a noise injection-noise reduction process and a data decomposition process on the target image label data to obtain a plurality of the multimodal samples, including:

[0021] Obtain geographic information data and terrain surface data of the current study area;

[0022] Performing a noise injection-noise reduction process on the target image label data through the geographic information data and the terrain surface data to obtain image-label joint data;

[0023] The image-label joint data is decomposed to obtain a plurality of the multimodal samples.

[0024] Optionally, the obtaining of geographic information data and terrain surface data of the current study area includes:

[0025] Acquire basic digital elevation model data of the current study area, and obtain the terrain surface data according to the basic digital elevation model data, wherein the terrain surface data includes slope data, aspect data and mountain shadow data;

[0026] According to the remote sensing multimodal semantic segmentation data set, geographic metadata and time step data are obtained, the geographic metadata and the time step data are linearly embedded respectively, and the geographic information data are obtained by adding them.

[0027] Optionally, the performing a noise injection-noise reduction process on the target image label data through the geographic information data and the terrain surface data to obtain image-label joint data includes:

[0028] The geographic information data is used as condition information, and the terrain surface data is used as a supplement to the condition information. The target image label data is subjected to a noise injection-noise reduction process to obtain image-label joint data.

[0029] Optionally, the step of performing data decomposition on the image-label joint data to obtain a plurality of the multimodal samples includes:

[0030] The latter band in the image-label joint data is inversely binary encoded and restored to obtain a plurality of the multimodal samples, wherein C represents the number of categories of the image-label joint data and represents a logarithmic function with base 2.

[0031] In a second aspect, the present invention provides a multimodal sample generation device based on a geographic information conditional diffusion model, comprising:

[0032] An acquisition unit, configured to acquire a remote sensing multimodal semantic segmentation data set, wherein the remote sensing multimodal semantic segmentation data set includes multi-source images and semantic labels corresponding to the multi-source images;

[0033] A training unit, used for inputting the remote sensing multimodal semantic segmentation data set into a geographic information conditional diffusion model for training to obtain a plurality of multimodal samples;

[0034] Wherein, the geographic information conditional diffusion model includes a joint input module and a data mosaic module;

[0035] The multi-source images and semantic labels in the remote sensing multimodal semantic segmentation dataset are subjected to channel splicing processing through the joint input module to obtain target image label data;

[0036] The target image label data is subjected to a noise injection-noise reduction process and a data decomposition process through the data embedding module to obtain a plurality of the multimodal samples.

[0037] In a third aspect, the present invention provides an electronic device, including a memory and a processor;

[0038] The memory is used to store computer programs;

[0039] The processor is used to implement the multimodal sample generation method based on the geographic information conditional diffusion model as described in the first aspect when executing the computer program.

[0040] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the multimodal sample generation method based on the geographic information conditional diffusion model as described in the first aspect is implemented.

[0041] The beneficial effects of the multimodal sample generation method, device, electronic device and storage medium based on the geographic information conditional diffusion model of the present invention are as follows: the present invention first obtains a data set containing multiple remote sensing image modalities and their corresponding semantic labels. It can be understood that the image can be obtained by different sensors, such as optical images, hyperspectral data, radar data, etc., and the semantic label corresponds to the ground object category information of each pixel; the acquired remote sensing multimodal semantic segmentation data set is input into the geographic information conditional diffusion model for training, and the model includes a joint input module and a data mosaic module. The geographic information conditional diffusion model is intended to generate a data set using the geographic information conditional diffusion algorithm. Multiple multimodal samples, wherein the joint input module is responsible for channel splicing processing of multi-source images and semantic labels in remote sensing multimodal semantic segmentation data sets, integrating information from different input sources, and obtaining target image label data, so that the model can better understand image features and object category information; the data mosaic module is responsible for noise injection-noise reduction process and data decomposition processing of target image label data to generate multiple multimodal samples, and the noise injection-noise reduction process can help reduce interference information in the data, making the sample clearer and more accurate; data decomposition helps to extract different features from the original data and generate diversified sample data. Through the method of the present invention, the geographic information conditional diffusion model can learn object category information from the remote sensing multimodal semantic segmentation data set, and generate multiple multimodal samples in combination with the geographic information conditional diffusion algorithm, with multi-source images and semantic labels as input, and processed by the joint input module and the data mosaic module, so as to generate more diverse and rich sample data, provide more comprehensive information and more choices for remote sensing image annotation, solve the problem that remote sensing image annotation data samples usually lack text description information, and achieve low-cost and high-efficiency generation of data samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of a flow chart of a multimodal sample generation method based on a geographic information conditional diffusion model according to an embodiment of the present invention;

[0043] Figure 2 A schematic diagram of the structure of a geographic information conditional diffusion model according to an embodiment of the present invention;

[0044] Figure 3 The figure is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be interpreted as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0046] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0047] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to"; the term "based on" means "based at least in part on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments"; the term "optionally" means "optional embodiments". The relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0048] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0049] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

[0050] like Figure 1 As shown, an embodiment of the present invention provides a multimodal sample generation method based on a geographic information conditional diffusion model, comprising:

[0051] Step S1, obtaining a remote sensing multimodal semantic segmentation data set, wherein the remote sensing multimodal semantic segmentation data set includes multi-source images and semantic labels corresponding to the multi-source images.

[0052] Specifically, a data set containing multiple remote sensing image modalities and their corresponding semantic labels is obtained. It can be understood that the images can be acquired by different sensors, such as optical images, hyperspectral data, radar data, etc., and the semantic labels correspond to the ground object category information of each pixel.

[0053] Step S2: input the remote sensing multimodal semantic segmentation data set into a geographic information conditional diffusion model for training to obtain a plurality of multimodal samples.

[0054] Specifically, this embodiment, through the geographic information conditional diffusion model, can use multi-source images and semantic labels to perform channel stitching processing to generate multiple multimodal samples, thereby enriching data diversity, improving the richness and diversity of samples, and enhancing the generalization ability of the model.

[0055] Step S3, wherein the geographic information conditional diffusion model includes a joint input module and a data mosaic module;

[0056] The multi-source images and semantic labels in the remote sensing multimodal semantic segmentation dataset are subjected to channel splicing processing through the joint input module to obtain target image label data;

[0057] The target image label data is subjected to a noise injection-noise reduction process and a data decomposition process through the data embedding module to obtain a plurality of the multimodal samples.

[0058] Specifically, the data mosaic module in this embodiment can perform noise injection-noise reduction and data decomposition processing on the target image label data, optimize data quality, and improve the availability and effectiveness of samples. This embodiment can better integrate and process multimodal data according to geographic information conditions, improve the model's understanding and expression capabilities of geographic information and surface features, and provide a more effective solution for information processing in a specific geographic environment. The multimodal sample generation method based on the geographic information conditional diffusion model of this embodiment can effectively improve data diversity, optimize data quality, and improve the model's understanding and utilization capabilities of geographic information and multimodal data, making the model more suitable for information processing and analysis in a specific geographic environment. In the process of generating images, adding random noise to the model input helps to improve the robustness and generalization ability of the model, helping the model to better capture subtle features in the data and making the model more interpretable. It is understandable that in actual application, the noise adding method includes but is not limited to introducing random disturbances into the input data, such as Gaussian noise, impulse noise or mask noise, etc., all of which aim to enable the model to better learn the distribution of the data, thereby generating more diverse and realistic images; after the image is generated, the generated image needs to be denoised to reduce the impact of the noise introduced in the generation process on the image quality, improve the clarity and quality of the generated image, and make it more consistent with the visual characteristics of the real world. It is understandable that the denoising method includes but is not limited to the use of neural network models, filters or denoising algorithms, all of which can effectively remove noise from the image and improve the visual quality of the image.

[0059] Specifically, the joint input module of this embodiment can perform channel stitching processing on the multi-source images and semantic labels in the remote sensing multimodal semantic segmentation dataset, thereby realizing the integration of multi-source data, enabling the model to obtain rich information from different angles and improve the variation and diversity of the data; through the data mosaic module, the target image label data is subjected to noise injection-noise reduction process and data decomposition processing, which can enhance the robustness of the data, reduce the interference of noise on model modeling, and improve the generalization ability of the model. Through this embodiment, samples with a variety of diversity features can be generated, which can better simulate the diversity of actual remote sensing images, improve the diversity and complexity of training data, and provide better conditions for the training of deep learning models.

[0060] This embodiment first obtains a data set containing multiple remote sensing image modalities and their corresponding semantic labels. It can be understood that the images can be obtained by different sensors, such as optical images, hyperspectral data, radar data, etc., and the semantic labels correspond to the ground object category information of each pixel; the obtained remote sensing multimodal semantic segmentation data set is input into the geographic information conditional diffusion model for training, and the model includes a joint input module and a data mosaic module. The geographic information conditional diffusion model aims to generate multiple multimodal samples using the geographic information conditional diffusion algorithm, wherein the joint input module is responsible for channel splicing processing of multi-source images and semantic labels in the remote sensing multimodal semantic segmentation data set, integrating information from different input sources, and obtaining target image label data, so that the model can better understand image features and ground object category information; the data mosaic module is responsible for performing noise injection-noise reduction process and data decomposition processing on the target image label data to generate multiple multimodal samples, and the noise injection-noise reduction process can help reduce interference information in the data, making the sample clearer and more accurate; data decomposition helps to extract different features from the original data and generate diversified sample data. Through the method of this embodiment, the geographic information conditional diffusion model can learn the category information of land objects from the remote sensing multimodal semantic segmentation data set, and generate multiple multimodal samples in combination with the geographic information conditional diffusion algorithm. It takes multi-source images and semantic labels as input, and uses the joint input module and the data mosaic module for processing, so as to generate more diverse and rich sample data, provide more comprehensive information and more choices for remote sensing image annotation, solve the problem that remote sensing image annotation data samples usually lack text description information, and achieve low-cost and high-efficiency generation of data samples.

[0061] Optionally, the acquiring of a remote sensing multimodal semantic segmentation dataset includes:

[0062] Obtain the image of the current study area and the mine classification and annotation data of the current study area;

[0063] According to the current study area image and the mine sub-classification annotation data, a plurality of the multi-source images and a plurality of the semantic labels are obtained;

[0064] The remote sensing multimodal semantic segmentation dataset is obtained according to the multiple multi-source images and the multiple semantic labels.

[0065] Specifically, this embodiment obtains multi-source images and semantic labels by acquiring images of the current study area and mine classification annotation data, so that the model can fully understand the information of different spectral bands, and at the same time, combined with the semantic information of the objects, provide the model with richer data features, including spectral, spatial and semantic information, which helps to improve the recognition and interpretation capabilities of the deep learning model for the objects. This embodiment uses a multimodal semantic segmentation data set to better mine the information of the objects, including their spatial distribution and semantic information, and provide more accurate data support for the segmentation and recognition tasks of mine objects, thereby improving the accurate classification and recognition capabilities of mine objects. The method of this embodiment can cope with the complex and diverse characteristics of remote sensing data, provide more comprehensive and diverse data features for the deep learning model, overcome the limitations of traditional single bands or specific images, better utilize multimodal information for object recognition and segmentation, and improve the performance of the deep learning model. This provides a data basis for generating more effective sample data.

[0066] Optionally, the step of performing channel stitching processing on multi-source images and semantic labels in the remote sensing multimodal semantic segmentation dataset through the joint input module to obtain target image label data includes:

[0067] Performing standardization processing on the multi-source images to obtain a standardized image;

[0068] Mapping the discrete label value corresponding to the semantic label into a binary vector representation to obtain a binary label;

[0069] The standardized image and the binary label are concatenated in the channel dimension to obtain target image label data.

[0070] Specifically, this embodiment standardizes the image data range, that is, standardizes the multi-source images to obtain a standardized image with a pixel range between [0, 1], and maps the semantic labels to binary vectors, thereby overcoming the problem of lack of descriptive information in the remote sensing image annotation data and improving the processing ability of the deep learning model for remote sensing data, wherein the range of the standardized image data helps to improve the stability of the training, slow down the impact of different data ranges, and make it easier for the model to grasp the main features of the data, increase the learning speed, and thus improve the sample generation efficiency; mapping the semantic labels to binary vectors enables the model to better understand the specific ground object information at the pixel level, provides richer information, and improves the model's recognition and segmentation accuracy of the ground object; through channel splicing, the standardized image and the binary label are spliced ​​in the channel dimension to provide richer training data and improve the generalization ability and accuracy of the deep learning model. The method of this embodiment improves the processing effect of the deep learning model on the remote sensing image and reduces the problem of lack of descriptive information in the annotation data by standardizing and vectorizing the remote sensing image and the semantic label.

[0071] Optionally, the data mosaic module performs a noise injection-noise reduction process and a data decomposition process on the target image label data to obtain a plurality of the multimodal samples, including:

[0072] Obtain geographic information data and terrain surface data of the current study area;

[0073] Performing a noise injection-noise reduction process on the target image label data through the geographic information data and the terrain surface data to obtain image-label joint data;

[0074] The image-label joint data is decomposed to obtain a plurality of the multimodal samples.

[0075] Specifically, the noise injection-noise reduction process of this embodiment can remove noise and interference in the target image label data, and improve the accuracy and reliability of the data; the data decomposition processing can decompose the image-label joint data into multiple multimodal samples, effectively enriching the diversity and complexity of the data, and providing more data selection and application possibilities. By processing the target image label data through geographic information data and terrain surface data, the relevance and practicality of the data can be improved, making the data more applicable and valuable. Multimodal samples can provide more comprehensive data information, enhance the robustness and generalization ability of the model, and make the model more suitable for different application scenarios and environmental conditions. This embodiment performs noise injection-noise reduction process and data decomposition processing on the target image label data through the data mosaic module, which can effectively improve data quality, enrich data samples, enhance data value and enhance model robustness, thereby realizing efficient and accurate generation of labeled sample data. This embodiment has significant advantages and application prospects.

[0076] Optionally, the obtaining of geographic information data and terrain surface data of the current study area includes:

[0077] Acquire basic digital elevation model data of the current study area, and obtain the terrain surface data according to the basic digital elevation model data, wherein the terrain surface data includes slope data, aspect data and mountain shadow data;

[0078] According to the remote sensing multimodal semantic segmentation data set, geographic metadata and time step data are obtained, the geographic metadata and the time step data are linearly embedded respectively, and the geographic information data are obtained by adding them.

[0079] Specifically, this embodiment obtains terrain surface data, including slope data, aspect data, and hill shadow data, by acquiring basic digital elevation model data, thereby enriching the dimension of geographic information, providing more terrain features, and enhancing the diversity and expression ability of the data. The geographic metadata and time step data are obtained using a remote sensing multimodal semantic segmentation data set, and they are linearly embedded and added to obtain geographic information data, thereby improving the relevance of the data and making the geographic information data more practical and reliable. By acquiring and integrating geographic information data and terrain surface data, the quality and integrity of the data can be improved, the value of the data can be increased, and more abundant information and possibilities can be provided for subsequent data processing and analysis. Obtaining rich geographic information data and terrain surface data helps to improve the expressive power of the model, so that the model can better capture the characteristics and laws of geographic space changes, and enhance the prediction and analysis capabilities of the model.

[0080] Optionally, the performing a noise injection-noise reduction process on the target image label data through the geographic information data and the terrain surface data to obtain image-label joint data includes:

[0081] The geographic information data is used as condition information, and the terrain surface data is used as a supplement to the condition information. The target image label data is subjected to a noise injection-noise reduction process to obtain image-label joint data.

[0082] Specifically, geographic information data and terrain surface data provide richer background information and contextual environment, which can help remove noise and interference in the target image label data and improve the effect and accuracy of the noise injection-denoising process; using geographic information data as conditional information and terrain surface data as a supplement to the conditional information can enhance the correlation between the target image label data and the geographic environment, help remove noise that is inconsistent with the geographic information, and reduce incorrect annotations and irrelevant information in the data; in the process of noise removal, the supplement of geographic information data and terrain surface data can help retain useful information in the target image label data, especially information related to the geographic environment, and improve the fidelity and practicality of the data after the noise injection-denoising process; by adding conditional constraints on geographic information data and terrain surface data, the model can be trained to better understand and utilize geographic environment information, thereby improving the robustness and generalization ability of the model.

[0083] Optionally, the step of performing data decomposition on the image-label joint data to obtain a plurality of the multimodal samples includes:

[0084] The latter band in the image-label joint data is inversely binary encoded and restored to obtain a plurality of the multimodal samples, wherein C represents the number of categories of the image-label joint data and represents a logarithmic function with base 2.

[0085] Specifically, by performing reverse binary coding on the image-label joint data for restoration, we can obtain multimodal samples of multiple different bands. These samples have different characteristics, enriching the diversity of the data and helping to improve the expressiveness and application value of the data. After obtaining multiple multimodal samples, we can better understand and explain the data characteristics and changes, explore the inherent laws and characteristics of the data, and improve the interpretability of the data and the understanding of the application field. Multiple multimodal samples can be used for model training, which can increase the amount of training data for the model, improve the robustness and generalization ability of the model, and make the model more suitable for different application scenarios. After obtaining multiple multimodal samples, we can improve the effective utilization of data, meet the needs of different application scenarios, and improve the flexibility and applicability of data.

[0086] In one embodiment, the multi-source image and the binary-encoded semantic label are jointly input to the encoder and spliced ​​at the input front channel. Since the semantic segmentation label in this embodiment is also in tif format, it is directly spliced ​​with the multi-source image along the channel (for example, the number of channels of the original multi-source image is 4, and the number of channels of the semantic label is generally 1, but in this embodiment, the label is mapped into a vector using binary encoding. If there are 8 categories, the label is mapped into a data format with a channel number of 3. The number of channels after splicing is 4+3=7, and the diffusion generates 7-channel image data. The first 4 channels are retained as the generated image, and the last three channels are reversed binary encoded to restore the label of 1 channel. In this way, the image and label are generated at the same time, and after being encoded by the encoder, vector addition is performed with the noise, and then the noise is input into the noise injection-noise reduction process for noise injection-noise reduction process processing, combined with Figure 2 As shown in the figure, the input of the upper diagram in the noise injection-denoising process uses the geographic metadata of the satellite image (such as longitude and latitude, year, etc.) as conditional information to guide the generation of the image. The geographic metadata and time step are linearly embedded and then added. The symbol in the upper left corner represents "vector addition". The longitude and latitude information obtained from the remote sensing image metadata are characters, which will be encoded into vector form through word embedding, and then the vectors are added; the time step is essential in the diffusion model, and it is also encoded into vector form, added to the encoded metadata, and jointly guides the diffusion step; the input of the lower diagram in the noise injection-denoising process uses the terrain surface data extracted from the basic DEM data (basic digital elevation model data) as a supplement to the conditional information, and the image-label joint data generated by the noise injection-denoising process is processed multiple times, and then it is decomposed and the latter The bands are inversely binary encoded (C represents the number of categories) to restore, thereby generating multi-source image-semantic label pairs (multi-modal samples), where Figure 2 The lower right corner indicates the cyclic execution of the noise injection-denoising process (the noise injection-denoising process is carried out step by step according to the time step setting, rather than being processed all at once. For example, if the time step is set to 1000, the noise injection-denoising process will be executed 1000 times to ensure the quality and diversity of the generated image).

[0087] An embodiment of the present invention provides a multimodal sample generation device based on a geographic information conditional diffusion model, comprising:

[0088] An acquisition unit, configured to acquire a remote sensing multimodal semantic segmentation data set, wherein the remote sensing multimodal semantic segmentation data set includes multi-source images and semantic labels corresponding to the multi-source images;

[0089] A training unit, used for inputting the remote sensing multimodal semantic segmentation data set into a geographic information conditional diffusion model for training to obtain a plurality of multimodal samples;

[0090] Wherein, the geographic information conditional diffusion model includes a joint input module and a data mosaic module;

[0091] The multi-source images and semantic labels in the remote sensing multimodal semantic segmentation dataset are subjected to channel splicing processing through the joint input module to obtain target image label data;

[0092] The target image label data is subjected to a noise injection-noise reduction process and a data decomposition process through the data embedding module to obtain a plurality of the multimodal samples.

[0093] The multimodal sample generation device based on the geographic information conditional diffusion model of this embodiment is used to implement the multimodal sample generation method based on the geographic information conditional diffusion model as described above. Its advantages over the prior art are the same as the advantages of the multimodal sample generation method based on the geographic information conditional diffusion model over the prior art, and will not be repeated here.

[0094] like Figure 3 As shown, an electronic device 300 provided by an embodiment of the present invention includes a memory 310 and a processor 320; the memory 310 is used to store a computer program; the processor 320 is used to implement the multimodal sample generation method based on the geographic information conditional diffusion model as described above when executing the computer program.

[0095] In other words, an electronic device 300 includes a memory 310 and a processor 320 coupled to the memory 310; the memory 310 is configured to store a computer program; and the processor 320 is configured to perform the following operations when executing the computer program:

[0096] Acquire a remote sensing multimodal semantic segmentation dataset, wherein the remote sensing multimodal semantic segmentation data includes multi-source images and semantic labels corresponding to the multi-source images;

[0097] Inputting the remote sensing multimodal semantic segmentation data set into a geographic information conditional diffusion model for training to obtain a plurality of multimodal samples;

[0098] Wherein, the geographic information conditional diffusion model includes a joint input module and a data mosaic module;

[0099] The multi-source images and semantic labels in the remote sensing multimodal semantic segmentation dataset are subjected to channel splicing processing through the joint input module to obtain target image label data;

[0100] The target image label data is subjected to a noise injection-noise reduction process and a data decomposition process through the data embedding module to obtain a plurality of the multimodal samples.

[0101] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the multimodal sample generation method based on the geographic information conditional diffusion model as described above is implemented.

[0102] In other words, a non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor performs the following operations:

[0103] Acquire a remote sensing multimodal semantic segmentation dataset, wherein the remote sensing multimodal semantic segmentation data includes multi-source images and semantic labels corresponding to the multi-source images;

[0104] Inputting the remote sensing multimodal semantic segmentation data set into a geographic information conditional diffusion model for training to obtain a plurality of multimodal samples;

[0105] Wherein, the geographic information conditional diffusion model includes a joint input module and a data mosaic module;

[0106] The multi-source images and semantic labels in the remote sensing multimodal semantic segmentation dataset are subjected to channel splicing processing through the joint input module to obtain target image label data;

[0107] The target image label data is subjected to a noise injection-noise reduction process and a data decomposition process through the data embedding module to obtain a plurality of the multimodal samples.

[0108] An electronic device 300 that can be used as a server or client of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device 300 is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device 300 can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present invention described and / or required herein.

[0109] The electronic device 300 includes a computing unit, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) or a computer program loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for the operation of the device can also be stored. The computing unit, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0110] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc. In the present application, the unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the present invention. In addition, each functional unit in each embodiment of the present invention can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0111] Although the present invention is disclosed as above, the protection scope of the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the protection scope of the present invention.

Claims

1. A multimodal sample generation method based on a geographic information conditional diffusion model, characterized in that: include: Acquire a remote sensing multimodal semantic segmentation dataset, wherein the remote sensing multimodal semantic segmentation data includes multi-source images and semantic labels corresponding to the multi-source images; Inputting the remote sensing multimodal semantic segmentation data set into a geographic information conditional diffusion model to obtain a plurality of multimodal samples; Wherein, the geographic information conditional diffusion model includes a joint input module and a data mosaic module; The multi-source images and semantic labels in the remote sensing multimodal semantic segmentation dataset are subjected to channel splicing processing through the joint input module to obtain target image label data; The target image label data is subjected to a noise injection-noise reduction process and a data decomposition process through the data embedding module to obtain a plurality of the multimodal samples.

2. The multimodal sample generation method based on the geographic information conditional diffusion model according to claim 1 is characterized in that: The step of obtaining a remote sensing multimodal semantic segmentation dataset comprises: Obtain the image of the current study area and the mine classification and annotation data of the current study area; According to the current study area image and the mine sub-classification annotation data, a plurality of the multi-source images and a plurality of the semantic labels are obtained; The remote sensing multimodal semantic segmentation dataset is obtained according to the multiple multi-source images and the multiple semantic labels.

3. The multimodal sample generation method based on the geographic information conditional diffusion model according to claim 2 is characterized in that: The method of performing channel splicing processing on the multi-source images and semantic labels in the remote sensing multimodal semantic segmentation dataset through the joint input module to obtain target image label data includes: Performing standardization processing on the multi-source images to obtain a standardized image; Mapping the discrete label value corresponding to the semantic label into a binary vector representation to obtain a binary label; The standardized image and the binary label are concatenated in the channel dimension to obtain target image label data.

4. The multimodal sample generation method based on the geographic information conditional diffusion model according to claim 1 is characterized in that: The data embedding module performs a noise injection-noise reduction process and a data decomposition process on the target image label data to obtain a plurality of the multimodal samples, including: Obtain geographic information data and terrain surface data of the current study area; Performing a noise injection-noise reduction process on the target image label data through the geographic information data and the terrain surface data to obtain image-label joint data; The image-label joint data is decomposed to obtain a plurality of the multimodal samples.

5. The multimodal sample generation method based on the geographic information conditional diffusion model according to claim 4 is characterized in that: The acquisition of geographic information data and terrain surface data of the current study area includes: Acquire basic digital elevation model data of the current study area, and obtain the terrain surface data according to the basic digital elevation model data, wherein the terrain surface data includes slope data, aspect data and mountain shadow data; According to the remote sensing multimodal semantic segmentation data set, geographic metadata and time step data are obtained, the geographic metadata and the time step data are linearly embedded respectively, and the geographic information data are obtained by adding them.

6. The multimodal sample generation method based on the geographic information conditional diffusion model according to claim 4 is characterized in that: The step of performing a noise injection-noise reduction process on the target image label data by using the geographic information data and the terrain surface data to obtain image-label joint data includes: The geographic information data is used as condition information, and the terrain surface data is used as a supplement to the condition information. The target image label data is subjected to a noise injection-noise reduction process to obtain image-label joint data.

7. The multimodal sample generation method based on the geographic information conditional diffusion model according to claim 4 is characterized in that: Decomposing the image-label joint data to obtain a plurality of multimodal samples includes: The image-label joint data is then The bands are inversely binary encoded and restored to obtain a plurality of the multimodal samples, wherein C represents the number of categories of the image-label joint data, and log2 represents a logarithmic function with base 2.

8. A multimodal sample generation device based on a geographic information conditional diffusion model, characterized in that: include: An acquisition unit, configured to acquire a remote sensing multimodal semantic segmentation data set, wherein the remote sensing multimodal semantic segmentation data set includes multi-source images and semantic labels corresponding to the multi-source images; A training unit, used for inputting the remote sensing multimodal semantic segmentation data set into a geographic information conditional diffusion model for training to obtain a plurality of multimodal samples; Wherein, the geographic information conditional diffusion model includes a joint input module and a data mosaic module; The multi-source images and semantic labels in the remote sensing multimodal semantic segmentation dataset are subjected to channel splicing processing through the joint input module to obtain target image label data; The target image label data is subjected to a noise injection-noise reduction process and a data decomposition process through the data embedding module to obtain a plurality of the multimodal samples.

9. An electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the multimodal sample generation method based on the geographic information conditional diffusion model as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the multimodal sample generation method based on the geographic information conditional diffusion model as described in any one of claims 1 to 7 is implemented.

Images