Remote sensing image scene generation method, device, storage medium and equipment
By adding random noise to the generation of remote sensing image scenes and using the remote sensing image scene generation model to predict noise, the problem of poor generation effect in the existing technology is solved, efficient and natural generation of remote sensing image scenes is achieved, and the generation effect and efficiency are improved. .
Patent Information
- Application Number
- CN202410402730.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-03
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-04-03
AI Technical Summary
The existing remote sensing image scene generation methods lack interpretability, ineffective feature extraction methods and low processing efficiency, and cannot directly synthesize natural and continuous remote sensing image scenes, which limits its development and application.
By determining the description information, adding random initialization noise to the area to be generated, and noise prediction is performed based on the remote sensing image scene generation model, peeling the target prediction noise to generate a remote sensing image scene, and improving the accuracy of segmentation or classification using diffusion algorithm and Transformer model.
A remote sensing image scene with a reasonable visual structure and similar texture to the surroundings is generated, which improves the generation effect without sacrificing the quality and clarity of the image, achieving more efficient scene generation.
Smart Images

Figure CN118506174B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of remote sensing image processing and computer vision, and in particular to a remote sensing image scene generation method, device, storage medium and equipment. Background Art
[0002] Remote sensing image scene generation is the use of computer science and geographic information system technology, using remote sensing images, map data, terrain data and other geographic data to create a simulated geographic environment. Remote sensing image scene generation can be used to simulate natural landforms, urban layout, transportation networks, emergency management and other fields for geographic analysis, planning and visualization, and plays an irreplaceable role in resource surveys, environmental monitoring, urban planning and other fields. By simulating real-world remote sensing image scenes, it helps researchers and decision makers better understand and analyze geospatial data, thereby supporting intelligent decision-making and planning.
[0003] However, the complexity, diversity and variability of remote sensing image scenes make remote sensing image scene generation face complex challenges, including knowledge embedding, scene synthesis and realistic modeling. First, the perception and understanding of remote sensing image scenes need to consider the spatial and temporal characteristics of geospatial data, which increases the difficulty of remote sensing image scene generation. Secondly, remote sensing image scene generation involves multiple disciplines, including geography, computer science and artificial intelligence, which also increases the complexity of remote sensing image scene generation technology. Finally, traditional remote sensing image scene generation methods have some problems, such as lack of interpretability, invalid feature extraction methods and low processing efficiency, which cannot directly synthesize natural and continuous remote sensing image scenes, limiting the development and application of remote sensing image scenes. Summary of the invention
[0004] In view of this, the present invention provides a remote sensing image scene generation method, device, storage medium and equipment, which can solve the technical problem of poor remote sensing image scene generation effect.
[0005] According to one aspect of the present invention, a remote sensing image scene generation method is provided, the method comprising:
[0006] Determine description information describing the remote sensing image scene generation result, and determine the remote sensing image to be generated according to the description information, wherein the remote sensing image scene generation result is used to cover the to-be-generated area of the to-be-generated remote sensing image to obtain a target remote sensing image;
[0007] Adding random initialization noise to the area to be generated to obtain the remote sensing image to be generated after the random initialization noise;
[0008] Based on the remote sensing image scene generation model, under the constraint of the description information, noise prediction is performed on the remote sensing image to be generated after the random initialization noise to obtain the target prediction noise, and the target prediction noise is stripped from the remote sensing image to be generated after the random initialization noise to generate the remote sensing image scene generation result, wherein the description information includes at least one of target content description knowledge and surrounding image information of the remote sensing image scene generation result.
[0009] Preferably, the determining of description information describing the remote sensing image scene generation result, and determining the remote sensing image to be generated according to the description information, comprises:
[0010] If the description information describing the remote sensing image scene generation result is only the target content description knowledge, determining the generation range of the remote sensing image to be generated;
[0011] If the description information is only the surrounding image information, or if the description information is the target content description knowledge and the surrounding image information, then the to-be-generated area of the remote sensing image to be generated is determined, wherein the to-be-generated remote sensing image is marked with the to-be-generated area, and the non-to-be-generated area of the remote sensing image to be generated is the surrounding image information.
[0012] Preferably, before performing noise prediction on the remote sensing image to be generated after the random initialization noise based on the remote sensing image scene generation model under the constraint of the description information, the method further comprises:
[0013] Constructing a training set, wherein the training set includes remote sensing images to be trained and content description knowledge describing the remote sensing images to be trained;
[0014] An initial remote sensing image scene generation model is constructed according to a diffusion algorithm, and the initial remote sensing image scene generation model is trained using the training set to obtain a remote sensing image scene generation model.
[0015] Preferably, the using the training set to train the initial remote sensing image scene generation model to obtain the remote sensing image scene generation model includes:
[0016] Adding a first random real noise of a preset step to all positions in the remote sensing image to be trained to obtain a first real noise remote sensing image;
[0017] Inputting the first real noise remote sensing image and the content description knowledge into the initial remote sensing image scene generation model to perform noise prediction to obtain predicted noise;
[0018] A first loss value between the first random real noise and the predicted noise is calculated, and training is performed iteratively to adjust the parameters of the initial remote sensing image scene generation model until the first loss value is less than a first preset threshold, thereby obtaining a remote sensing image scene generation model.
[0019] Preferably, the using the training set to train the initial remote sensing image scene generation model to obtain the remote sensing image scene generation model includes:
[0020] Adding the second random real noise of the step before the preset step to all positions in the remote sensing image to be trained to obtain a second real noise remote sensing image;
[0021] The predicted noise remote sensing image is obtained by subtracting the predicted noise from the first real noise remote sensing image, and a second loss value between the second real noise remote sensing image and the predicted noise remote sensing image is calculated. The training is iteratively performed to adjust the parameters of the initial remote sensing image scene generation model until the second loss value is less than the first preset threshold value, thereby obtaining the remote sensing image scene generation model.
[0022] Preferably, the method further comprises:
[0023] Obtain remote sensing image rectifier;
[0024] The predicted noisy remote sensing image is scored using the remote sensing image rectifier to obtain a first scoring result, a second scoring result is calculated based on the first scoring result and the second loss value, training is iterated, and the parameters of the initial remote sensing image scene generation model are adjusted until the second scoring result is greater than a second preset threshold and the second loss value is less than the first preset threshold, so as to obtain the remote sensing image scene generation model.
[0025] Preferably, after determining the remote sensing image to be generated according to the description information, the method further comprises:
[0026] compressing the remote sensing image to be generated to obtain a target representation vector;
[0027] Adding random initialization noise to the to-be-generated region of the target representation vector to obtain a representation vector after random initialization noise;
[0028] Based on the remote sensing image scene generation model, under the constraint of the description information, noise prediction is performed on the representation vector after the random initialization noise to obtain target prediction noise, the target prediction noise is stripped from the representation vector after the random initialization noise to obtain a final representation vector, and the final representation vector is restored to the remote sensing image scene generation result.
[0029] Preferably, the method further comprises:
[0030] Using the remote sensing image scene generation result to cover the area to be generated to obtain a target remote sensing image;
[0031] Reviewing the generation accuracy of the target remote sensing image, selecting the target remote sensing image whose generation accuracy is greater than a third preset threshold as a newly added remote sensing image to be trained, and determining new content description knowledge for describing the newly added remote sensing image to be trained;
[0032] Adding the newly added remote sensing image to be trained and the corresponding newly added content description knowledge to the training set to obtain an updated training set, and using the updated training set to train the remote sensing image scene generation model to obtain an updated remote sensing image scene generation model;
[0033] Based on the updated remote sensing image scene generation model, under the constraint of the description information, noise prediction is performed on the remote sensing image to be generated after the random initialization noise to obtain an updated target prediction noise, the updated target prediction noise is stripped from the remote sensing image to be generated after the random initialization noise to generate an updated remote sensing image scene generation result, and the updated remote sensing image scene generation result is used to cover the area to be generated to obtain an updated target remote sensing image.
[0034] Preferably, the step of stripping the target prediction noise from the remote sensing image to be generated after the random initialization noise comprises:
[0035] The target prediction noise is stripped from the remote sensing image to be generated after the random initialization noise using a sampling algorithm, wherein the sampling algorithm includes at least one of DDPM, DDIM, DPM-Solver, DPM-Solver++, and UniPC.
[0036] Preferably, the performing of noise prediction on the remote sensing image to be generated after the random initialization noise under the constraint of the description information comprises:
[0037] If the description information contains the target content description knowledge, the target content description knowledge is semantically parsed using Transformer based on the remote sensing image scene generation model, and noise prediction is performed on the remote sensing image to be generated after random initialization of noise according to the parsing result.
[0038] Preferably, the using the training set to train the initial remote sensing image scene generation model to obtain the remote sensing image scene generation model includes:
[0039] Compress the remote sensing image to be trained to obtain a representation vector to be trained, and add random noise to the representation vector to be trained once using a preset noise addition formula to obtain a noise representation vector to be trained, wherein the preset noise addition formula is: In the formula, z t is the noise representation vector to be trained at time step t, is the cumulative noise control parameter of step t, z0 is the representation vector to be trained, ∈ represents the random noise, which is the standard Gaussian distribution noise;
[0040] The representation vector to be trained and the content description knowledge are input into the initial remote sensing image scene generation model, the initial remote sensing image scene generation model is used to perform noise prediction to obtain predicted noise, a third loss value of the noise representation vector to be trained and the predicted noise is calculated, and the parameters of the initial remote sensing image scene generation model are adjusted until the third loss value is less than a fourth preset threshold value to obtain a remote sensing image scene generation model.
[0041] According to another aspect of the present invention, a remote sensing image scene generation device is provided, the device comprising:
[0042] A determination module, used to determine description information describing a remote sensing image scene generation result, and determine a remote sensing image to be generated according to the description information, wherein the remote sensing image scene generation result is used to cover a to-be-generated area of the to-be-generated remote sensing image to obtain a target remote sensing image;
[0043] An adding module is used to add random initialization noise to the area to be generated to obtain the remote sensing image to be generated after the random initialization noise;
[0044] The first prediction module is used to perform noise prediction on the remote sensing image to be generated after the random initialization noise based on the remote sensing image scene generation model under the constraint of the description information to obtain target prediction noise, and to strip the target prediction noise from the remote sensing image to be generated after the random initialization noise to generate the remote sensing image scene generation result, wherein the description information includes at least one of target content description knowledge and surrounding image information of the remote sensing image scene generation result.
[0045] According to another aspect of the present invention, there is provided a storage medium having a computer program stored thereon, wherein the program implements the above-mentioned remote sensing image scene generation method when executed by a processor.
[0046] According to another aspect of the present invention, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the above-mentioned remote sensing image scene generation method when executing the program.
[0047] By means of the above technical scheme, a remote sensing image scene generation method, device, storage medium and equipment provided by the present invention can first determine the description information describing the remote sensing image scene generation result, and determine the remote sensing image to be generated according to the description information, wherein the remote sensing image scene generation result is used to cover the to-be-generated area of the to-be-generated remote sensing image to obtain the target remote sensing image; then add random initialization noise to the to-be-generated area to obtain the to-be-generated remote sensing image after the random initialization noise; finally, based on the remote sensing image scene generation model, under the constraint of the description information, perform noise prediction on the to-be-generated remote sensing image after the random initialization noise to obtain the target prediction noise, and strip the target prediction noise from the to-be-generated remote sensing image after the random initialization noise to generate the remote sensing image scene generation result, wherein the description information includes at least one of the target content description knowledge and the surrounding image information of the remote sensing image scene generation result. In summary, under the constraint of description information, the remote sensing image scene generation model performs noise prediction in the direction of the remote sensing image generation result. When the description information is the target content description knowledge, the remote sensing image scene generation model can obtain the geographic entities in the remote sensing image corresponding to the text concept. When the description information is the surrounding image information of the remote sensing image scene generation result, the remote sensing image scene generation model can use the contextual information and interaction between pixels to improve the accuracy of segmentation or classification, that is, learn the surrounding environment characteristics, thereby obtaining a remote sensing image scene generation result with a reasonable visual structure and similar texture to the surrounding. The generation effect is good and the image quality and clarity are not sacrificed.
[0048] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation on the local application. In the drawings:
[0050] Figure 1 A schematic diagram of a process of generating a remote sensing image scene provided by an embodiment of the present invention is shown;
[0051] Figure 2 A schematic diagram showing a flow chart of another remote sensing image scene generation method provided by an embodiment of the present invention;
[0052] Figure 3 A schematic diagram of the structure of a remote sensing image scene generation device provided by an embodiment of the present invention is shown;
[0053] Figure 4 A schematic diagram showing the structure of another remote sensing image scene generation device provided by an embodiment of the present invention is shown;
[0054] Figure 5 Two types of binary mask images provided by embodiments of the present invention are shown;
[0055] Figure 6 A schematic diagram of generating a remote sensing image scene (using a binary mask) provided by an embodiment of the present invention is shown;
[0056] Figure 7 A comparison diagram of the effects of remote sensing image scene generation results with and without adding knowledge provided by an embodiment of the present invention is shown;
[0057] Figure 8 A comparison diagram of the effects of remote sensing image scene generation results under different knowledge intensities provided by an embodiment of the present invention is shown;
[0058] Fig. 9 A schematic diagram of remote sensing image scene generation (using text) provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0059] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present invention can be combined with each other without conflict.
[0060] This embodiment provides a remote sensing image scene generation method, such as Figure 1 As shown, the method includes:
[0061] 101. Determine description information for describing a remote sensing image scene generation result, and determine a remote sensing image to be generated according to the description information, wherein the remote sensing image scene generation result is used to cover a to-be-generated area of the to-be-generated remote sensing image to obtain a target remote sensing image.
[0062] It should be noted that remote sensing images are images of the earth's surface information collected through remote sensing technology, which contain a lot of important geographical, environmental and resource information. Due to the characteristics of high efficiency, high resolution and wide coverage, remote sensing images are widely used in geological exploration, agricultural management, urban planning, environmental monitoring, hydrology, forestry management and other fields. Remote sensing image scenes refer to regional complexes with specific structures and functions formed by the interconnection and interaction of various natural and human elements within a certain time and space range. Remote sensing image scenes can be used in agricultural production, urban planning, environmental protection, public safety and other fields, and have important application value.
[0063] In this embodiment, the entire area or part of the area of the remote sensing image to be generated needs to be regenerated, and the area to be regenerated is the area to be generated of the remote sensing image to be generated. It should be noted that the remote sensing image to be generated may or may not have pixel information before being regenerated. For example, Fig. 9 As shown, when the description information only has the target content description knowledge, then the remote sensing image to be generated does not have pixel information before it is generated, and the remote sensing image to be generated is a pixel-free image of the same size as the remote sensing image scene generation result (wherein the same size as the remote sensing image scene generation result is the generation range of the remote sensing image to be generated), and all its areas need to be regenerated. For example, when the description information only has the surrounding image information of the remote sensing image scene generation result, then the remote sensing image to be generated has pixel information before it is regenerated, and part of its area needs to be regenerated. For example, when the description information includes the target content description knowledge and the surrounding image information of the remote sensing image scene generation result, then the remote sensing image to be generated has pixel information before it is regenerated, and all or part of its area needs to be regenerated.
[0064] As an implementation method, if the area to be generated is the entire area of the remote sensing image to be generated, the area to be generated does not need to be marked, and the remote sensing image scene generation model will automatically identify the entire area as the area to be generated. If the area to be generated is a partial area of the remote sensing image to be generated, the area to be generated needs to be marked. The marking method can be to generate a binary mask image corresponding to the remote sensing image to be generated. The binary mask image is the same size as the remote sensing image to be generated, and only the area to be generated and the area not to be generated are marked on it, such as Figure 5 As shown, two types of binary mask images are disclosed. The left image uses a white mask to mark the area to be generated (the black part is the area not to be generated and does not need to be generated), and the right image uses a black mask to mark the area to be generated (the white part is the area not to be generated and does not need to be generated). Figure 6 As shown, the remote sensing image to be generated and the binary mask image are taken as a whole, so that the area to be generated is marked in the remote sensing image to be generated.
[0065] It should be noted that in addition to using Figure 5 In addition to the rectangle shown in the figure, you can also use circles, random smearing, etc. to select the area, without limitation here.
[0066] 102. Add random initialization noise to the area to be generated to obtain the remote sensing image to be generated after the random initialization noise.
[0067] It should be noted that noise appears in the form of useless information, disturbing the observable information of the image. Therefore, for this embodiment, the function of adding random initialization noise to the area to be generated is to blur the pixel information in the area to be generated. The random initialization noise added to the area to be generated can be added gradually to obtain the remote sensing image to be generated after the random initialization noise, or can be added in one step to obtain the remote sensing image to be generated after the random initialization noise, which is not limited here.
[0068] 104. Based on the remote sensing image scene generation model, under the constraint of the description information, noise prediction is performed on the remote sensing image to be generated after the random initialization noise to obtain target prediction noise, and the target prediction noise is stripped from the remote sensing image to be generated after the random initialization noise to generate the remote sensing image scene generation result, wherein the description information includes at least one of target content description knowledge and surrounding image information of the remote sensing image scene generation result.
[0069] Among them, the target content description knowledge may include: text (words or sentences may be used), semantic segmentation map, depth map, etc., or it may include a combination of two or more, which is not limited here. For example, the target content description knowledge is text, for example, the target content description knowledge is text and semantic segmentation map.
[0070] Taking the description information including both target content description knowledge and surrounding image information of remote sensing image scene generation results as an example, what is obtained is a remote sensing image scene generation result with reasonable visual structure and similar texture to the surrounding under the constraint of target content description knowledge (for example, the target content description knowledge is a house, that is, the remote sensing image scene generation result should be a house, not other things, such as vehicles, animals, etc.), such as Figure 7 As shown, a comparison diagram of the effects of remote sensing image scene generation results with and without adding knowledge is disclosed, wherein steps 1 to 40 are the process of gradually stripping noise, and the black frame represents the area to be generated. Taking step 40 as an example, it can be seen that the remote sensing image scene generation result of the area to be generated is better when image knowledge is added than when no knowledge is added.
[0071] The reason why the remote sensing image scene generation model can generate an image with a visually reasonable structure and texture similar to the surrounding area is that the remote sensing image scene generation model is built on the diffusion algorithm. The diffusion algorithm can propagate and generate sample labels in an iterative manner by using the similarity or correlation between pixels, thereby utilizing the contextual information and interactions between pixels to improve the accuracy of segmentation or classification.
[0072] The reason why the remote sensing image scene generation model can recognize the semantic information conveyed by the target content description knowledge is that the target content description knowledge is semantically parsed through Transformer. The parsed result is recognizable by the remote sensing image scene generation model. If the target content description knowledge is text, the modality unification between the text and the remote sensing image to be generated can be achieved through Transformer (while the target content description knowledge is a semantic segmentation map and a depth map, and the remote sensing image to be generated is also a map, so there is no need for modality unification), thereby realizing the coordinated control and directional generation of data of different modalities.
[0073] Specifically, under the constraint of the description information, noise prediction is performed on the remote sensing image to be generated after the random initialization noise, including: if the description information contains the target content description knowledge, based on the remote sensing image scene generation model, using Transformer to perform semantic analysis on the target content description knowledge, and according to the analysis result, noise prediction is performed on the remote sensing image to be generated after the random initialization noise.
[0074] Among them, in each cycle (for example, one step of the time step is one cycle), 1. noise prediction is performed to generate target prediction noise, and 2. target prediction noise is stripped from the remote sensing image to be generated after random initialization noise (in fact, target prediction noise is stripped from the area to be generated after adding random initialization noise). After a preset number of iterations, the remote sensing image scene generation result is obtained. As for stripping the target prediction noise from the remote sensing image to be generated after random initialization noise, it includes: stripping the target prediction noise from the remote sensing image to be generated after random initialization noise by using a sampling algorithm, wherein the sampling algorithm includes at least one of DDPM, DDIM, DPM-Solver, DPM-Solver++, and UniPC, which is not limited here. Noise stripping through the sampling algorithm can achieve the preset effect faster, significantly reduce the consumption of computing resources, and improve the generation efficiency of the remote sensing image scene generation result.
[0075] It should be noted that, based on the remote sensing image scene generation model, under the constraint of the description information, before performing noise prediction on the remote sensing image to be generated after the random initialization noise, the method also includes: constructing a training set, wherein the training set includes the remote sensing image to be trained and content description knowledge describing the remote sensing image to be trained; building an initial remote sensing image scene generation model according to the diffusion algorithm, and using the training set to train the initial remote sensing image scene generation model to obtain the remote sensing image scene generation model.
[0076] Specifically, the training set includes the remote sensing image to be trained and the content description knowledge describing the remote sensing image to be trained. For example, the type of the content description knowledge of the remote sensing image to be trained is text (in this case, the training set includes data of two modalities, text and image). Specifically, for example, a remote sensing image contains a stadium surrounded by a large number of densely distributed buildings and trees. It should be noted that in order to increase the sample size during training, the training set includes the entire remote sensing image to be trained and the content description knowledge describing the entire remote sensing image to be trained.
[0077] It should be noted that the specific process of the diffusion algorithm is: first, an initial label is assigned to each pixel, and then the similarity or correlation between each pixel and its surrounding pixels is iteratively calculated, and the pixel label is generated based on this information. This process will continue until the label propagation converges or reaches a predetermined stopping condition.
[0078] Specifically, the subjects when calculating the loss value are different, and two implementation methods are taken as examples to specifically illustrate how to use the training set to train the initial remote sensing image scene generation model to obtain the remote sensing image scene generation model:
[0079] It should be noted that the role of adding random initialization noise in step 103 of the embodiment is different from the role of adding random noise in the following two implementations. The role of adding random initialization noise in step 103 of the embodiment is to blur the pixel information of the area to be generated, while the role of adding random noise in the following two implementations is to add a label, which is a real value. The noise prediction obtained by using the initial remote sensing image scene generation model is a predicted value. The loss between the real value and the predicted value is calculated, and the model parameters are adjusted to make the loss value small enough, that is, the predicted value is close to the real value. At this time, the model is trained and the remote sensing image scene generation model is obtained. It is just that for the first implementation, the main body of calculating the loss value is between noise and noise, and for the second implementation, the main body of calculating the loss value is between remote sensing images and remote sensing images.
[0080] The first implementation method is to add a preset step of first random real noise to all positions in the remote sensing image to be trained to obtain a first real noise remote sensing image; input the first real noise remote sensing image and content description knowledge into the initial remote sensing image scene generation model for noise prediction to obtain predicted noise; calculate the first loss value between the first random real noise and the predicted noise, iterate training, and adjust the parameters of the initial remote sensing image scene generation model until the first loss value is less than a first preset threshold to obtain a remote sensing image scene generation model.
[0081] The second implementation method is to add a second random real noise of one step before a preset step to all positions in the remote sensing image to be trained to obtain a second real noise remote sensing image; calculate the first real noise remote sensing image minus the predicted noise to obtain a predicted noise remote sensing image, and calculate the second loss value between the second real noise remote sensing image and the predicted noise remote sensing image.
[0082] Among them, it should be noted that the preset step refers to the time step of the noise being the preset value. If the preset value is t, the time step is t, referred to as t step. Specifically, the first random real noise is the random real noise of the preset step, for example, t step random real noise, and the first real noise remote sensing image is obtained by gradually adding t step random real noise to all positions in the remote sensing image to be trained; the second random real noise is the random real noise of the step before the preset step, for example, t-1 step random real noise, and the second real noise remote sensing image is obtained by gradually adding t-1 step random real noise to all positions in the remote sensing image to be trained. It should be noted that the relationship between the first real noise remote sensing image and the second real noise remote sensing image is that the first real noise remote sensing image is obtained by adding one step of random real noise to the second real noise remote sensing image, and the second real noise remote sensing image is obtained by subtracting one step of random real noise from the first real noise remote sensing image. Therefore, for the second implementation, the predicted noise remote sensing image is obtained by subtracting the predicted noise from the first real noise remote sensing image. The subtraction action is taken one step back so that the noise of the predicted noise remote sensing image and the second real noise remote sensing image are at the same step. Only then can the second loss value between the second real noise remote sensing image and the predicted noise remote sensing image be calculated.
[0083] Preferably, for the second implementation, according to the different model training stop conditions, two implementations are taken as examples for specific description. In the first implementation, iterative training is performed to adjust the parameters of the initial remote sensing image scene generation model until the second loss value is less than the first preset threshold value, so as to obtain the remote sensing image scene generation model. In the second implementation, a remote sensing image rectifier is obtained; the predicted noisy remote sensing image is scored using the remote sensing image rectifier to obtain a first scoring result, a second scoring result is calculated based on the first scoring result and the second loss value, iterative training is performed to adjust the parameters of the initial remote sensing image scene generation model until the second scoring result is greater than the second preset threshold value and the second loss value is less than the first preset threshold value, so as to obtain the remote sensing image scene generation model.
[0084] Among them, the remote sensing image rectifier will score the result predicted by the initial remote sensing image scene generation model, that is, the predicted noise remote sensing image, during the training process of the initial remote sensing image scene generation model, and then feed back the score to the initial remote sensing image scene generation model. The specific feedback method is to use the remote sensing image rectifier to score the predicted noise remote sensing image on the generation effect level (preset the relationship between the generation effect level and the score), obtain a first scoring result, calculate the second scoring result according to the first scoring result and the second loss value, and feed back the second scoring result to the initial remote sensing image scene generation model. Among them, calculating the second scoring result according to the first scoring result and the second loss value may include: calculating the product of the first scoring result and the second loss value to obtain the second scoring result, which is not limited here.
[0085] Preferably, as an implementation mode, when the above-mentioned embodiment reaches the training stop condition (i.e., the first loss value is less than the first preset threshold; the second loss value is less than the first preset threshold; the second scoring result is greater than the second preset threshold, and the second loss value is less than the first preset threshold), the remote sensing image scene generation model is not obtained directly, but by adjusting the model parameters, a plurality of selected remote sensing image scene generation models are obtained, and these selected remote sensing image scene generation models all meet their respective training stop conditions, and these selected remote sensing image scene generation models are monitored, and the one with the best performance is selected as the remote sensing image scene generation model.
[0086] Preferably, as an implementation mode, after obtaining the remote sensing image scene generation result, the method further includes: using the remote sensing image scene generation result to cover the area to be generated to obtain the target remote sensing image; reviewing the generation accuracy of the target remote sensing image, and screening out the target remote sensing image with a generation accuracy greater than a third preset threshold as the newly added remote sensing image to be trained; determining the new content description knowledge for describing the newly added remote sensing image to be trained, adding the newly added remote sensing image to be trained and the corresponding new content description knowledge to the training set to obtain an updated training set, and using the updated training set to train the remote sensing image scene generation model to obtain an updated remote sensing image scene generation model; based on the updated remote sensing image scene generation model, under the constraint of the description information, performing noise prediction on the remote sensing image to be generated after the random initialization noise, obtaining an updated target prediction noise, stripping the updated target prediction noise from the remote sensing image to be generated after the random initialization noise to generate an updated remote sensing image scene generation result, and using the updated remote sensing image scene generation result to cover the area to be generated to obtain an updated target remote sensing image.
[0087] Among them, the generation accuracy of the target remote sensing image is manually reviewed, and then the target remote sensing images with a generation accuracy greater than the third preset threshold are selected. The purpose is to add training samples to the original training set to obtain an updated training set, and then use the updated training set to train the remote sensing image scene generation model. The obtained updated remote sensing image scene generation model has a better generation effect. The training process is the same as the above-mentioned use of the training set to train the initial remote sensing image scene generation model. The process of using the updated remote sensing image scene generation model to obtain an updated target remote sensing image is the same as the above-mentioned process of using the remote sensing image scene generation model to obtain the target remote sensing image, which will not be repeated here.
[0088] The present invention provides a remote sensing image scene generation method, device, storage medium and equipment, which can first determine description information describing the remote sensing image scene generation result, determine the remote sensing image to be generated according to the description information, wherein the remote sensing image scene generation result is used to cover the to-be-generated area of the to-be-generated remote sensing image to obtain the target remote sensing image; then add random initialization noise to the to-be-generated area to obtain the to-be-generated remote sensing image after the random initialization noise; finally, based on a remote sensing image scene generation model, under the constraint of the description information, perform noise prediction on the to-be-generated remote sensing image after the random initialization noise to obtain the target prediction noise, and strip the target prediction noise from the to-be-generated remote sensing image after the random initialization noise to generate the remote sensing image scene generation result, wherein the description information includes at least one of target content description knowledge and surrounding image information of the remote sensing image scene generation result. In summary, under the constraint of description information, the remote sensing image scene generation model performs noise prediction in the direction of the remote sensing image generation result. When the description information is the target content description knowledge, the remote sensing image scene generation model can obtain the geographic entities in the remote sensing image corresponding to the text concept. When the description information is the surrounding image information of the remote sensing image scene generation result, the remote sensing image scene generation model can use the contextual information and interaction between pixels to improve the accuracy of segmentation or classification, that is, learn the surrounding environment characteristics, thereby obtaining a remote sensing image scene generation result with a reasonable visual structure and similar texture to the surrounding. The generation effect is good and the image quality and clarity are not sacrificed.
[0089] Further, as a refinement and extension of the specific implementation of the above embodiment, in order to fully illustrate the specific implementation process of this embodiment, another remote sensing image scene generation method is provided, such as Figure 2 As shown, the method includes:
[0090] 201. Determine description information for describing a remote sensing image scene generation result, and determine a remote sensing image to be generated according to the description information, wherein the remote sensing image scene generation result is used to cover a to-be-generated area of the to-be-generated remote sensing image to obtain a target remote sensing image.
[0091] For this embodiment, the description information describing the remote sensing image scene generation result is determined, and the remote sensing image to be generated is determined according to the description information, including: if the description information describing the remote sensing image scene generation result is only the target content description knowledge, then the generation range of the remote sensing image to be generated is determined; if the description information is only the surrounding image information, or if the description information is the target content description knowledge and the surrounding image information, then the to-be-generated area of the remote sensing image to be generated is determined, wherein the to-be-generated area of the remote sensing image to be generated is marked with the to-be-generated area, and the non-to-be-generated area of the remote sensing image to be generated is the surrounding image information. Among them, the same size as the remote sensing image scene generation result is the generation range of the remote sensing image to be generated.
[0092] As a preferred implementation manner, before determining the target content description knowledge that describes the remote sensing image scene generation result, the knowledge strength level corresponding to the remote sensing image scene generation result is determined, and then the target content description knowledge that describes the remote sensing image scene generation result is determined based on the knowledge strength level, wherein the knowledge strength level is related to the detail level of the target content description knowledge, that is, the higher the knowledge strength level, the more detailed the target content description knowledge, and the better the generation effect of the remote sensing image scene generation result.
[0093] For example, there are three levels of knowledge intensity. If the knowledge intensity level corresponding to the remote sensing image scene generation result is determined to be the lowest level, then the corresponding target content description knowledge can be houses. If the knowledge intensity level corresponding to the remote sensing image scene generation result is determined to be an intermediate level, then the corresponding target content description knowledge can be residential houses. If the knowledge intensity level corresponding to the remote sensing image scene generation result is determined to be the highest level, then the corresponding target content description knowledge can be dense residential houses.
[0094] As another preferred implementation, there is no need to manually determine the knowledge strength level corresponding to the remote sensing image scene generation result as above, but to set a certain knowledge strength factor such as 10 or a knowledge strength factor range such as [1,30] and input it into the remote sensing image scene generation model. The larger the knowledge strength factor, the more the remote sensing image scene generation model obeys the input target content description knowledge. At this time, the manual workload can be reduced, and the target content description knowledge can be standardized. The output results of the remote sensing image scene generation model will not be different due to different descriptions of the target content description knowledge by different people.
[0095] like Figure 8 As shown, a comparison diagram of the effects of remote sensing image scene generation results under different knowledge intensities is disclosed. The larger the knowledge intensity factor, the higher the knowledge intensity level, and the more the remote sensing image scene generation model obeys the input target content description knowledge. The black frame is the area to be generated. Steps 1 to 30 are the process of gradually stripping off the noise. Taking step 30 as an example, it can be seen that compared with the knowledge intensity factor 1, the knowledge intensity factor 30 has a better effect on the remote sensing image scene generation result of the area to be generated, and is more in line with the input target content description knowledge.
[0096] 202. Compress the remote sensing image to be generated to obtain a target representation vector, add random initialization noise to the to-be-generated region of the target representation vector, and obtain a representation vector after random initialization noise.
[0097] For this embodiment, as an implementation method, the diffusion model, as a generative model, needs to learn complex high-dimensional data distribution, which places extremely high demands on model parameters and computing resources. Since there is a lot of redundant information in high-dimensional data, the computing and storage overhead can be reduced by removing this redundant information. In order to greatly speed up the generation speed, the generation process is migrated from the image space to a lower-dimensional latent space. Specifically, before adding random initialization noise, the remote sensing image to be generated is compressed, wherein the compression method may include: using a variational autoencoder to compress the remote sensing image to be generated, and obtaining the target representation vector in the latent space:
[0098] z 目标 =ε(x)
[0099] In the formula, x is the remote sensing image to be generated, ε is the encoder of the variational autoencoder, and z 目标 is the target representation vector in the latent space, where the dimension is reduced compared to the remote sensing image to be generated.
[0100] Then we represent the vector z to the target 目标 Add random initialization noise to obtain the representation vector after random initialization noise (for example, add t steps, then from z 目标 Get z t ), wherein the random initialization noise may be added all at once or gradually, which is not limited here.
[0101] It should be noted that, based on the different orders of compression and adding random initialization noise, compression can be performed first and then random initialization noise can be added, or random initialization noise can be added first and then compression can be performed, which is not limited here. For adding random initialization noise first and then compression, specifically: first, random initialization noise is added to the area to be generated to obtain the remote sensing image to be generated after random initialization noise, and then, the remote sensing image to be generated after random initialization noise is compressed to obtain the representation vector after random initialization noise.
[0102] 203. Based on the remote sensing image scene generation model, under the constraint of the description information, perform noise prediction on the representation vector after the random initialization noise to obtain target prediction noise, remove the target prediction noise from the representation vector after the random initialization noise to obtain a final representation vector, and restore the final representation vector to the remote sensing image scene generation result, wherein the description information includes at least one of target content description knowledge and surrounding image information of the remote sensing image scene generation result.
[0103] For this embodiment, noise prediction is performed in latent space, and target prediction noise is stripped in latent space. Compared with the high-dimensional space of remote sensing images, the speed is improved. After the final representation vector is obtained, the final representation vector is restored to obtain the remote sensing image scene generation result.
[0104] Specifically, taking compression first and then adding random initialization noise as an example, the target prediction noise is stripped from the representation vector after random initialization noise. Taking DDPM as an example, the stripping method is:
[0105]
[0106] In the formula, z t-1 is the representation vector after stripping the target prediction noise from the representation vector after randomly initializing the noise in t steps, α t +β t =1 is the noise control parameter, is the cumulative noise control parameter in t steps, z t is the representation vector after t steps of random initialization noise, ∈ θ (z t ,t) is the target prediction noise for the prediction step t,∈ 初 Represents random initialization noise, which is standard Gaussian distribution noise and obeys standard normal distribution.
[0107] When the preset number of times is reached, t-1 To represent the final vector, use Represents the final representation vector.
[0108] Then, the final representation vector is restored to obtain the remote sensing image scene generation result, which is expressed by the formula:
[0109]
[0110] In the formula, is the final representation vector, is the decoder of the variational autoencoder, Generate results for remote sensing imagery scenes.
[0111] It should be noted that the process of training to obtain the remote sensing image scene generation model is: constructing a training set, wherein the training set includes the remote sensing image to be trained and the content description knowledge that describes the remote sensing image to be trained, building an initial remote sensing image scene generation model according to the diffusion algorithm, compressing the remote sensing image to be trained to obtain the representation vector to be trained, using a preset noise addition formula to add random noise to the representation vector to be trained once, to obtain the noise representation vector to be trained (random noise can also be added first and then compressed, which will not be repeated here), inputting the representation vector to be trained and the content description knowledge into the initial remote sensing image scene generation model, using the initial remote sensing image scene generation model to perform noise prediction to obtain predicted noise, calculating the third loss value of the noise representation vector to be trained and the predicted noise, adjusting the parameters of the initial remote sensing image scene generation model until the third loss value is less than the fourth preset threshold, the model training is completed, and the remote sensing image scene generation model is obtained.
[0112] Among them, the formula for compressing the remote sensing image to be trained to obtain the representation vector to be trained is:
[0113] z0=ε(x0)
[0114] Where x0 is the remote sensing image to be trained, ε is the encoder of the variational autoencoder, and z0 is the representation vector to be trained in the latent space, where the dimension is reduced compared to the remote sensing image to be trained.
[0115] In the training process, random noise is added to the representation vector to be trained. The noise is added to the representation vector to be trained at a specific time step t using a preset noise adding formula, that is, it is added in place at one time, rather than gradually added in place, thereby speeding up the training speed. The preset noise adding formula is expressed as:
[0116]
[0117] In the formula, z t is the noise representation vector to be trained at time step t, is the cumulative noise control parameter of t steps, z0 is the representation vector to be trained, ∈ represents random noise, which is standard Gaussian distribution noise and obeys standard normal distribution.
[0118] During the training process, the third loss value of the noise representation vector to be trained and the predicted noise is calculated, and the formula is:
[0119]
[0120] In the formula, z is the representation vector in the latent space, y is the conditional input (i.e., content description knowledge), ∈ represents random noise, which is a standard Gaussian distribution noise and obeys the normal distribution, t represents the time step, ∈ θ () is the initial remote sensing image scene generation model to be trained, ∈θ (z t ,t,τ θ (y)) is the prediction noise, z t represents the noise representation vector to be trained at time step t, τ θ () is a conditional controller (i.e., a content description knowledge controller, which can control the knowledge intensity level, i.e., the knowledge intensity factor).
[0121] 204. Cover the to-be-generated area with the remote sensing image scene generation result to obtain a target remote sensing image.
[0122] Experimental results show that this research method is superior to existing models in terms of remote sensing scene generation quality, efficiency and realism, and can generate natural, continuous and realistic target remote sensing images.
[0123] The present invention provides a remote sensing image scene generation method, device, storage medium and equipment, which optimizes scene description information and integrates the relationship between complex geographical elements into model training, so that the remote sensing image scene generation model has the ability to accurately recognize remote sensing images. Specifically, by collecting remote sensing images to be trained, experts generate corresponding content description knowledge for the remote sensing images to be trained one by one (improving the accuracy and interpretability of the input description), such as text, to construct a remote sensing image to be trained-description text data set, and use a diffusion algorithm to train the data set to obtain a remote sensing image scene generation model, so that it has the ability to accurately recognize remote sensing images and can establish a connection between geographical entities in remote sensing images and text concepts, thereby realizing the input of corresponding text or images, and the model can understand the user's intention; and because the diffusion algorithm can The similarity or correlation between pixels is used in an iterative manner to propagate and generate sample labels, and then the contextual information and interaction between pixels are used to improve the accuracy of segmentation or classification, so as to generate remote sensing images with a visually reasonable structure and texture similar to the surroundings, and realize the intelligent generation of remote sensing image scenes; at the same time, in order to improve the generation efficiency, this patent uses a low-rank representation to train the diffusion model, that is, to migrate the generation process from the image space to a lower-dimensional latent space, and perform noise prediction and noise stripping in the latent space, which can greatly speed up the generation speed; in order to further improve the generation effect, this patent scores the generation effect of the remote sensing image generation results, uses high-scoring samples as training samples, updates them to the training set, and uses the updated training set to iteratively train the remote sensing image scene generation model to continuously improve the generation effect.
[0124] Further, as Figure 1 and Figure 2 The specific implementation of the method shown in the figure, the embodiment of the present invention provides a remote sensing image scene generation device, such as Figure 3 As shown, the device includes: a determination module 31, an adding module 32, and a first prediction module 33;
[0125] The determination module 31 may be used to determine description information describing the remote sensing image scene generation result, and determine the remote sensing image to be generated according to the description information, wherein the remote sensing image scene generation result is used to cover the to-be-generated area of the to-be-generated remote sensing image to obtain a target remote sensing image;
[0126] The first prediction module is used to perform noise prediction on the remote sensing image to be generated after the random initialization noise based on the remote sensing image scene generation model under the constraint of the description information to obtain target prediction noise, and to strip the target prediction noise from the remote sensing image to be generated after the random initialization noise to generate the remote sensing image scene generation result, wherein the description information includes at least one of target content description knowledge and surrounding image information of the remote sensing image scene generation result.
[0127] Correspondingly, in order to determine the description information that describes the remote sensing image scene generation result, the remote sensing image to be generated is determined according to the description information. The determination module 31 can be specifically used to determine the generation range of the remote sensing image to be generated if the description information that describes the remote sensing image scene generation result is only the target content description knowledge; if the description information is only the surrounding image information, or if the description information is the target content description knowledge and the surrounding image information, then determine the to-be-generated area of the remote sensing image to be generated, wherein the remote sensing image to be generated is marked with the to-be-generated area, and the non-to-be-generated area of the remote sensing image to be generated is the surrounding image information.
[0128] In a specific application scenario, a remote sensing image scene generation device, such as Figure 4 As shown, the device also includes: a first training module 30, which can be specifically used to construct a training set, wherein the training set includes the remote sensing image to be trained and the content description knowledge describing the remote sensing image to be trained; building an initial remote sensing image scene generation model according to the diffusion algorithm, and using the training set to train the initial remote sensing image scene generation model to obtain the remote sensing image scene generation model.
[0129] Correspondingly, in order to use the training set to train the initial remote sensing image scene generation model to obtain the remote sensing image scene generation model, the first training module 30 can also be used to add a preset step of first random real noise to all positions in the remote sensing image to be trained to obtain a first real noise remote sensing image; input the first real noise remote sensing image and the content description knowledge into the initial remote sensing image scene generation model for noise prediction to obtain predicted noise; calculate the first loss value between the first random real noise and the predicted noise, iterate training, and adjust the parameters of the initial remote sensing image scene generation model until the first loss value is less than a first preset threshold to obtain a remote sensing image scene generation model.
[0130] Correspondingly, in order to use the training set to train the initial remote sensing image scene generation model to obtain the remote sensing image scene generation model, the first training module 30 can also be used to add a second random real noise one step before the preset step to all positions in the remote sensing image to be trained to obtain a second real noise remote sensing image; calculate the first real noise remote sensing image minus the predicted noise to obtain a predicted noise remote sensing image, calculate the second loss value between the second real noise remote sensing image and the predicted noise remote sensing image, iterate the training, adjust the parameters of the initial remote sensing image scene generation model until the second loss value is less than the first preset threshold, and obtain the remote sensing image scene generation model.
[0131] Correspondingly, the first training module 30 can also be specifically used to obtain a remote sensing image rectifier; use the remote sensing image rectifier to score the predicted noise remote sensing image to obtain a first scoring result, calculate the second scoring result based on the first scoring result and the second loss value, iterate the training, and adjust the parameters of the initial remote sensing image scene generation model until the second scoring result is greater than the second preset threshold and the second loss value is less than the first preset threshold, so as to obtain the remote sensing image scene generation model.
[0132] In a specific application scenario, a remote sensing image scene generation device, such as Figure 4As shown, the device also includes: a second training module 34, which can be specifically used to use the remote sensing image scene generation result to cover the area to be generated to obtain a target remote sensing image; review the generation accuracy of the target remote sensing image, screen out the target remote sensing image with a generation accuracy greater than a third preset threshold as a newly added remote sensing image to be trained, and determine the new content description knowledge that describes the newly added remote sensing image to be trained; add the newly added remote sensing image to be trained and the corresponding new content description knowledge to the training set to obtain an updated training set, and use the updated training set to train the remote sensing image scene generation model to obtain an updated remote sensing image scene generation model; based on the updated remote sensing image scene generation model, under the constraint of the description information, perform noise prediction on the remote sensing image to be generated after the random initialization noise, obtain an updated target prediction noise, strip the updated target prediction noise from the remote sensing image to be generated after the random initialization noise to generate an updated remote sensing image scene generation result, and use the updated remote sensing image scene generation result to cover the area to be generated to obtain an updated target remote sensing image.
[0133] In a specific application scenario, a remote sensing image scene generation device, such as Figure 4 As shown, the device also includes: a second prediction module 35, which can be specifically used to compress the remote sensing image to be generated to obtain a target representation vector; add random initialization noise to the to-be-generated area of the target representation vector to obtain a representation vector after random initialization noise; based on the remote sensing image scene generation model, under the constraint of the description information, perform noise prediction on the representation vector after random initialization noise to obtain target prediction noise, strip the target prediction noise from the representation vector after random initialization noise to obtain a final representation vector, and restore the final representation vector to the remote sensing image scene generation result.
[0134] Correspondingly, in order to remove the target prediction noise from the remote sensing image to be generated after the random initialization noise, the first prediction module 33 can be specifically used to remove the target prediction noise from the remote sensing image to be generated after the random initialization noise by using a sampling algorithm, wherein the sampling algorithm includes at least one of: DDPM, DDIM, DPM-Solver, DPM-Solver++, and UniPC.
[0135] Correspondingly, in order to predict the noise of the remote sensing image to be generated after the random initialization noise is constrained by the description information, the first prediction module 33 can be specifically used to perform semantic analysis of the target content description knowledge based on the remote sensing image scene generation model using Transformer if the description information contains the target content description knowledge, and predict the noise of the remote sensing image to be generated after the random initialization noise according to the analysis result.
[0136] Correspondingly, in order to use the training set to train the initial remote sensing image scene generation model to obtain the remote sensing image scene generation model, the first training module 30 can also be used to compress the remote sensing image to be trained to obtain a representation vector to be trained, and add random noise to the representation vector to be trained once using a preset noise addition formula to obtain a noise representation vector to be trained, wherein the preset noise addition formula is: In the formula, z t is the noise representation vector to be trained at time step t, is the t-step cumulative noise control parameter, z0 is the representation vector to be trained, ∈ represents the random noise, which is standard Gaussian distribution noise; the representation vector to be trained and the content description knowledge are input into the initial remote sensing image scene generation model, the initial remote sensing image scene generation model is used to perform noise prediction to obtain predicted noise, the third loss value of the noise representation vector to be trained and the predicted noise is calculated, the parameters of the initial remote sensing image scene generation model are adjusted until the third loss value is less than the fourth preset threshold, and the remote sensing image scene generation model is obtained.
[0137] It should be noted that for other corresponding descriptions of the functional units involved in the remote sensing image scene generation device provided in this embodiment, reference can be made to Figure 1 to Figure 2 The corresponding description will not be repeated here.
[0138] Based on the above Figure 1 to Figure 2 The method shown in the embodiment, accordingly, also provides a storage medium, which can be volatile or non-volatile, and stores a computer program, which is executed by a processor to implement the above-mentioned Figure 1 to Figure 2 The remote sensing image scene generation method shown.
[0139] Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various implementation scenarios of the present invention.
[0140] Based on the above Figure 1 to Figure 2 The method shown and Figure 3 , Figure 4 In order to achieve the above-mentioned purpose, the embodiment of the present application further provides a computer device, which can be a personal computer, a server, a network device, etc. The computer device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above-mentioned Figure 1 and Figure 2 The remote sensing image scene generation method shown.
[0141] Optionally, the computer device may further include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.
[0142] Those skilled in the art will appreciate that the computer device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different component arrangements.
[0143] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the above-mentioned computer device and supports the operation of information processing programs and other software and / or programs. The network communication module is used to realize communication between the components inside the non-volatile storage medium, and communication with other hardware and software in the information processing entity device.
[0144] Through the description of the above implementation modes, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform, or by hardware.
[0145] The present invention provides a remote sensing image scene generation method, device, storage medium and equipment, which can first determine description information describing the remote sensing image scene generation result, determine the remote sensing image to be generated according to the description information, wherein the remote sensing image scene generation result is used to cover the to-be-generated area of the to-be-generated remote sensing image to obtain the target remote sensing image; then add random initialization noise to the to-be-generated area to obtain the to-be-generated remote sensing image after the random initialization noise; finally, based on a remote sensing image scene generation model, under the constraint of the description information, perform noise prediction on the to-be-generated remote sensing image after the random initialization noise to obtain the target prediction noise, and strip the target prediction noise from the to-be-generated remote sensing image after the random initialization noise to generate the remote sensing image scene generation result, wherein the description information includes at least one of target content description knowledge and surrounding image information of the remote sensing image scene generation result. In summary, under the constraint of description information, the remote sensing image scene generation model performs noise prediction in the direction of the remote sensing image generation result. When the description information is the target content description knowledge, the remote sensing image scene generation model can obtain the geographic entities in the remote sensing image corresponding to the text concept. When the description information is the surrounding image information of the remote sensing image scene generation result, the remote sensing image scene generation model can use the contextual information and interaction between pixels to improve the accuracy of segmentation or classification, that is, learn the surrounding environment characteristics, thereby obtaining a remote sensing image scene generation result with a reasonable visual structure and similar texture to the surrounding. The generation effect is good and the image quality and clarity are not sacrificed.
[0146] Those skilled in the art will appreciate that the accompanying drawings are only schematic diagrams of a preferred implementation scenario, and the modules or processes in the accompanying drawings are not necessarily necessary for implementing the present invention. Those skilled in the art will appreciate that the modules in the devices in the implementation scenario can be distributed in the devices of the implementation scenario according to the description of the implementation scenario, or can be changed accordingly and located in one or more devices different from the present implementation scenario. The modules of the above-mentioned implementation scenario can be combined into one module, or can be further split into multiple submodules.
[0147] The above serial numbers of the present invention are only for description and do not represent the advantages and disadvantages of the implementation scenarios. The above disclosure is only a few specific implementation scenarios of the present invention, but the present invention is not limited thereto, and any changes that can be thought of by those skilled in the art should fall within the protection scope of the present invention.
Claims
1. A remote sensing image scene generation method, characterized in that: The method comprises: Determine description information describing the remote sensing image scene generation result, and determine the remote sensing image to be generated according to the description information, wherein the remote sensing image scene generation result is used to cover the to-be-generated area of the to-be-generated remote sensing image to obtain a target remote sensing image; Adding random initialization noise to the area to be generated to obtain the remote sensing image to be generated after the random initialization noise; Based on the remote sensing image scene generation model, under the constraint of the description information, noise prediction is performed on the remote sensing image to be generated after the random initialization noise to obtain target prediction noise, and the target prediction noise is stripped from the remote sensing image to be generated after the random initialization noise to generate the remote sensing image scene generation result, wherein the description information includes at least one of target content description knowledge and surrounding image information of the remote sensing image scene generation result; The step of determining description information describing the remote sensing image scene generation result, and determining the remote sensing image to be generated according to the description information, comprises: If the description information describing the remote sensing image scene generation result is only the target content description knowledge, determining the generation range of the remote sensing image to be generated; If the description information is only the surrounding image information, or if the description information is the target content description knowledge and the surrounding image information, determining a to-be-generated area of the remote sensing image to be generated, wherein the to-be-generated remote sensing image is marked with the to-be-generated area, and a non-to-be-generated area of the to-be-generated remote sensing image is the surrounding image information; In the remote sensing image scene generation model, before performing noise prediction on the remote sensing image to be generated after random initialization of noise under the constraint of the description information, the method further includes: Constructing a training set, wherein the training set includes remote sensing images to be trained and content description knowledge describing the remote sensing images to be trained; An initial remote sensing image scene generation model is constructed according to a diffusion algorithm, and the initial remote sensing image scene generation model is trained using the training set to obtain a remote sensing image scene generation model.
2. The method according to claim 1, characterized in that The step of training the initial remote sensing image scene generation model using the training set to obtain the remote sensing image scene generation model includes: Adding a preset step of first random real noise to all positions in the remote sensing image to be trained to obtain a first real noise remote sensing image; Inputting the first real noise remote sensing image and the content description knowledge into the initial remote sensing image scene generation model to perform noise prediction to obtain predicted noise; A first loss value between the first random real noise and the predicted noise is calculated, and training is performed iteratively to adjust the parameters of the initial remote sensing image scene generation model until the first loss value is less than a first preset threshold, thereby obtaining a remote sensing image scene generation model.
3. The method according to claim 2, characterized in that The step of training the initial remote sensing image scene generation model using the training set to obtain the remote sensing image scene generation model includes: Adding the second random real noise of the step before the preset step to all positions in the remote sensing image to be trained to obtain a second real noise remote sensing image; The predicted noise remote sensing image is obtained by subtracting the predicted noise from the first real noise remote sensing image, and a second loss value between the second real noise remote sensing image and the predicted noise remote sensing image is calculated. The training is iteratively performed to adjust the parameters of the initial remote sensing image scene generation model until the second loss value is less than the first preset threshold value, thereby obtaining the remote sensing image scene generation model.
4. The method according to claim 3, characterized in that The method further comprises: Obtain remote sensing image rectifier; The predicted noisy remote sensing image is scored using the remote sensing image rectifier to obtain a first scoring result, a second scoring result is calculated based on the first scoring result and the second loss value, training is iterated, and the parameters of the initial remote sensing image scene generation model are adjusted until the second scoring result is greater than a second preset threshold and the second loss value is less than the first preset threshold, so as to obtain the remote sensing image scene generation model.
5. The method according to claim 1, characterized in that After determining the remote sensing image to be generated according to the description information, the method further includes: compressing the remote sensing image to be generated to obtain a target representation vector; Adding random initialization noise to the to-be-generated region of the target representation vector to obtain a representation vector after random initialization noise; Based on the remote sensing image scene generation model, under the constraint of the description information, noise prediction is performed on the representation vector after the random initialization noise to obtain target prediction noise, the target prediction noise is stripped from the representation vector after the random initialization noise to obtain a final representation vector, and the final representation vector is restored to the remote sensing image scene generation result.
6. The method according to claim 1, characterized in that The method further comprises: Using the remote sensing image scene generation result to cover the area to be generated to obtain a target remote sensing image; Reviewing the generation accuracy of the target remote sensing image, selecting the target remote sensing image whose generation accuracy is greater than a third preset threshold as a newly added remote sensing image to be trained, and determining new content description knowledge for describing the newly added remote sensing image to be trained; Adding the newly added remote sensing image to be trained and the corresponding newly added content description knowledge to the training set to obtain an updated training set, and using the updated training set to train the remote sensing image scene generation model to obtain an updated remote sensing image scene generation model; Based on the updated remote sensing image scene generation model, under the constraint of the description information, noise prediction is performed on the remote sensing image to be generated after the random initialization noise to obtain an updated target prediction noise, the updated target prediction noise is stripped from the remote sensing image to be generated after the random initialization noise to generate an updated remote sensing image scene generation result, and the updated remote sensing image scene generation result is used to cover the area to be generated to obtain an updated target remote sensing image.
7. The method according to claim 1, characterized in that The step of stripping the target prediction noise from the remote sensing image to be generated after the random initialization noise comprises: The target prediction noise is stripped from the remote sensing image to be generated after the random initialization noise using a sampling algorithm, wherein the sampling algorithm includes at least one of DDPM, DDIM, DPM-Solver, DPM-Solver++, and UniPC.
8. The method according to claim 1, characterized in that The step of performing noise prediction on the remote sensing image to be generated after the random initialization noise under the constraint of the description information includes: If the description information contains the target content description knowledge, the target content description knowledge is semantically parsed using Transformer based on the remote sensing image scene generation model, and noise prediction is performed on the remote sensing image to be generated after random initialization of noise according to the parsing result.
9. The method according to claim 1, characterized in that: The step of training the initial remote sensing image scene generation model using the training set to obtain the remote sensing image scene generation model includes: Compress the remote sensing image to be trained to obtain a representation vector to be trained, and add random noise to the representation vector to be trained once using a preset noise addition formula to obtain a noise representation vector to be trained, wherein the preset noise addition formula is: , where is the noise representation vector to be trained at time step t, is the cumulative noise control parameter for t steps, is the representation vector to be trained, represents the random noise, which is standard Gaussian distribution noise; The representation vector to be trained and the content description knowledge are input into the initial remote sensing image scene generation model, the initial remote sensing image scene generation model is used to perform noise prediction to obtain predicted noise, a third loss value of the noise representation vector to be trained and the predicted noise is calculated, and the parameters of the initial remote sensing image scene generation model are adjusted until the third loss value is less than a fourth preset threshold value to obtain a remote sensing image scene generation model.
10. A remote sensing image scene generation device, characterized in that: The device comprises: A determination module, used to determine description information describing a remote sensing image scene generation result, and determine a remote sensing image to be generated according to the description information, wherein the remote sensing image scene generation result is used to cover a to-be-generated area of the to-be-generated remote sensing image to obtain a target remote sensing image; An adding module is used to add random initialization noise to the area to be generated to obtain the remote sensing image to be generated after the random initialization noise; A first prediction module is used to perform noise prediction on the remote sensing image to be generated after the random initialization noise based on the remote sensing image scene generation model under the constraint of the description information to obtain target prediction noise, and to strip the target prediction noise from the remote sensing image to be generated after the random initialization noise to generate the remote sensing image scene generation result, wherein the description information includes at least one of target content description knowledge and surrounding image information of the remote sensing image scene generation result; The determination module is specifically used to determine the generation range of the remote sensing image to be generated if the description information describing the remote sensing image scene generation result is only the target content description knowledge; if the description information is only the surrounding image information, or if the description information is the target content description knowledge and the surrounding image information, determine the to-be-generated area of the remote sensing image to be generated, wherein the to-be-generated remote sensing image to be generated is marked with the to-be-generated area, and the non-to-be-generated area of the to-be-generated remote sensing image to be generated is the surrounding image information; The first training module is specifically used to construct a training set, wherein the training set includes the remote sensing image to be trained and the content description knowledge describing the remote sensing image to be trained; an initial remote sensing image scene generation model is built according to the diffusion algorithm, and the initial remote sensing image scene generation model is trained using the training set to obtain the remote sensing image scene generation model.
11. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the remote sensing image scene generation method according to any one of claims 1 to 9 is implemented.
12. A computer device comprising a memory, a processor, and a computer program stored in a storage medium and executable on the processor, characterized in that: When the processor executes the program, the remote sensing image scene generation method described in any one of claims 1 to 9 is implemented.
