Remote sensing image controllable generation method based on multi-modal information guidance

Abstract

The invention discloses a remote sensing image controllable generation method based on multi-modal information guidance. The method comprises the following steps: constructing a structured text with an object and spatial distribution information thereof; establishing mapping between the structured text and the image layout; according to the image layout prediction, obtaining a foreground object image matched with the layout prediction from a foreground object database, and encoding to obtain foreground object visual features; and synthesizing a remote sensing image based on image layout prediction and structured text description by taking the visual features of the foreground object as guidance. According to the method disclosed by the invention, the remote sensing image with the corresponding semantics can be generated according to the given condition information, and meanwhile, the diversity and complexity of the quantity, the category and the spatial distribution of the objects in the image can be ensured.

Description

Technical Field

[0001] The invention relates to a controllable generation method of a remote sensing image based on multi-modal information guidance, and belongs to the technical field of image synthesis. Background Art

[0002] With the rapid development of earth observation technology and imaging sensor performance, the size and resolution of remote sensing images have been significantly improved. This has led to a sharp increase in the number of parameters in the target extraction network, which in turn has put forward higher requirements on the size of the training data set. However, the acquisition and annotation process of remote sensing data is time-consuming and laborious, involving a large amount of manpower and material resources.

[0003] The diffusion model-based Vincent graph method has the ability to generate images based on text descriptions. In theory, it can be used for the synthesis or annotation of remote sensing data. However, in practice, its framework structure and training objectives are mostly designed for the synthesis of natural images (such as faces, animals, buildings, etc.), and are mainly used in games, painting, photography and other fields. Compared with natural images, objects in remote sensing images have large scale differences, fewer texture features, dense spatial distribution, and random object directions. Therefore, the data distribution of the two is significantly different, which makes it difficult for the existing diffusion model-based Vincent graph method to directly achieve the synthesis of remote sensing images.

[0004] In addition, although existing remote sensing image generation methods can guide the semantics of synthetic images with conditional information such as text descriptions, these methods still have obvious limitations in controlling the number, category and spatial position of objects in the output image. It is difficult to synthesize training samples in a targeted manner according to specific task requirements or model performance, which greatly reduces the value of synthetic images as training data and seriously restricts their application in remote sensing visual perception model training.

[0005] Therefore, it is necessary to conduct in-depth research on remote sensing image generation methods to solve the above problems. Summary of the invention

[0006] In order to overcome the above problems, in-depth research was conducted and a controllable generation method of remote sensing images guided by multimodal information was proposed, including:

[0007] S1. Construct structured text with information about objects and their spatial distribution;

[0008] S2, establish a mapping between structured text and image layout;

[0009] S3, according to the image layout, obtaining the foreground object image matching the layout prediction from the foreground object database, and encoding to obtain the visual features of the foreground object;

[0010] S4. Based on image layout prediction and structured text description, remote sensing images are synthesized with the guidance of the visual features of foreground objects.

[0011] In a preferred embodiment, the image is divided into a plurality of image blocks, and the relative positions of the image blocks where the objects are located are described to obtain structured text having information about the objects and their spatial distribution.

[0012] In a preferred embodiment, the structured text also includes a description of the number of objects in the image block.

[0013] In a preferred embodiment, the structured text has a plurality of sentences, including:

[0014] Describe the name and specific quantity of each type of object in order of quantity;

[0015] Describe the names and specific quantities of objects in the image block in the central area in order of quantity;

[0016] The names, specific quantities and spatial distribution of objects in other regional image blocks are described in order of quantity.

[0017] In a preferred embodiment, before describing the image, the number of objects is sorted by category, and several categories with the largest number of objects or categories with a number of objects greater than a threshold are selected as target categories. When describing, only the target categories and their specific numbers are described.

[0018] In a preferred embodiment, in S2, an autoregressive network with a Transformer architecture is used to learn the spatial position relationship between objects in the image by establishing a sequential dependency relationship between objects in the text.

[0019] In a preferred embodiment, in S3, the foreground object image is input into the visual encoder DINOv2, and the visual features of the foreground object are obtained through the visual encoder DINOv2.

[0020] In a preferred embodiment, in S4, a diffusion model is used to generate multiple instances to synthesize remote sensing images.

[0021] The present invention also provides an electronic device, comprising:

[0022] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform any of the above methods.

[0023] The present invention also provides a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute any of the above-mentioned methods.

[0024] The beneficial effects of the present invention include:

[0025] (1) Using the description text as a guide, a reasonable object layout is generated to suit the user's intention, which reduces the user's burden and improves the automation level of the image generation process;

[0026] (2) It has good controllability and can generate remote sensing images with corresponding semantics based on given conditional information, while ensuring that the number, category and spatial distribution of objects in the image are diverse and complex;

[0027] (3) Effectively generate high-fidelity remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic flow chart of a method for controllable generation of remote sensing images based on multimodal information guidance according to a preferred embodiment of the present invention;

[0029] Figure 2 A schematic diagram of the process of image segmentation and construction of structured text with information about objects and their spatial distribution in Example 1;

[0030] Figure 3 Schematic diagram of the image layout prediction and remote sensing image synthesis process in Example 1;

[0031] Figure 4 A mapping result between the structured text and the image layout is generated in Example 1;

[0032] Figure 5 This is the remote sensing image result generated in Example 1. DETAILED DESCRIPTION

[0033] The present invention will be further described in detail below through the accompanying drawings and embodiments. Through these descriptions, the characteristics and advantages of the present invention will become more clear and distinct.

[0034] The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise noted.

[0035] According to a controllable generation method of remote sensing images based on multimodal information guidance provided by the present invention, Figure 1 As shown, including:

[0036] S1. Construct structured text with information about objects and their spatial distribution;

[0037] S2, establish a mapping between structured text and image layout;

[0038] S3, according to the image layout prediction, obtaining the foreground object image matching the layout prediction from the foreground object database, and encoding to obtain the visual features of the foreground object;

[0039] S4. Based on image layout prediction and structured text description, remote sensing images are synthesized with the guidance of the visual features of foreground objects.

[0040] It is difficult for existing natural language control texts to achieve cross-modal semantic consistency between text and images, which has a great negative impact on the controllability of synthetic images. Analysis shows that the main reason for the above problem is that the existing control texts can only summarize the categories, attributes (such as color, shape, etc.) and other features of visual targets in remote sensing images, but it is difficult to clearly describe the number of objects and spatial distribution positions. To solve this problem, in the present invention, by constructing a structured text with information about objects and their spatial distribution, the spatial information of the target object can be efficiently introduced into the text, thereby improving the expressive power of the text description.

[0041] In a preferred embodiment, the image is divided into a plurality of image blocks, and the relative positions of the image blocks where the objects are located are described to obtain structured text having information about the objects and their spatial distribution.

[0042] More preferably, the structured text further includes a description of the number of objects in the image block.

[0043] In a preferred embodiment, the structured text has a plurality of sentences, including:

[0044] Describe the name and specific quantity of each type of object in order of quantity;

[0045] Describe the names and specific quantities of objects in the image block in the central area in order of quantity;

[0046] The names, specific quantities and spatial distribution of objects in other regional image blocks are described in order of quantity.

[0047] Preferably, the number of sentences in each structured text is the same, for example, 5.

[0048] According to the present invention, those skilled in the art can freely set the size of the central area image block according to actual needs. For example, the image is divided into K×K image blocks, and the K / 2×K / 2 image blocks located at the center of the image are taken as central area image blocks, and the remaining image blocks are named according to their relative positions compared to the central area, such as "upper area", "lower right area", etc.

[0049] Preferably, before describing the image, the number of objects is sorted by category, and several categories with the largest number of objects or categories with a number of objects greater than a threshold are selected as target categories. When describing, only the target categories and their specific numbers are described.

[0050] According to the present invention, when sorting the number of objects in a specific category, the specific setting value of the target category number or the object number threshold is not limited, and technical personnel in this field can freely set it according to actual needs, for example, selecting the three categories of objects with the largest number as the target categories.

[0051] Preferably, when the number of objects is greater than a set threshold, for example, when the number of objects is greater than 5, a unified quantifier is used to refer to it. The quantifier may be "a lot", "Many", etc., which is not limited in the present invention.

[0052] Preferably, when describing an image, three sentences are used to describe the three largest types of objects respectively. When there are less than three target classes, the description sentences are left blank to ensure that the total number of sentences in each structured text is the same.

[0053] According to the structured text construction method described in the present invention, the spatial distribution and relative position information of the main objects in the image can be expressed in the shortest possible language, and the semantic ambiguity is reduced as much as possible without using words to describe the precise spatial coordinates of the objects. It has the advantages of low computational complexity and high efficiency.

[0054] In a preferred embodiment, the existing large model can be used to construct the structured text.

[0055] For example, large language models such as BERT (Bidirectional Encoder Representations from Transformers) and LLaMA (Large Language Model Meta AI) can be used to directly output the large language models after fine-tuning them using remote sensing image datasets; or online multimodal large models such as GPT-4 (Generative Pre-trained Transformer-4) or Gemini can be used to obtain structured text by adjusting prompt words and conducting multiple rounds of conversations.

[0056] In S2, preferably, an autoregressive network with a Transformer architecture is used to learn the spatial position relationship between objects in the image by establishing a sequential dependency relationship between objects in the text, thereby achieving a mapping between structured text and image layout, and obtaining image layout prediction through mapping.

[0057] In a preferred embodiment, during the training of the Transformer architecture autoregressive network, the category of the object is first predicted {l i}(i=1,2,…,N, N is the total number of object categories) and the image block number to which the center point belongs {k j}(j=1,2,…,K 2 ), and then regress the specific object frame parameters (x, y, w, h) based on this.

[0058] The above-mentioned "coarse-to-fine" training strategy can reduce the difficulty of modeling cross-modal data mapping from text information to spatial semantics, and improve the efficiency and stability of regression network model training.

[0059] Furthermore, this method successfully achieves the transformation from descriptive text to image layout with cross-modal semantic consistency.

[0060] In S3, the foreground object database refers to a database consisting only of images of the objects themselves.

[0061] By removing the background of the object and retaining only the object itself, the background information is prevented from interfering with the subsequent generation process, and the domain differences of the existing generation models built on general image datasets in remote sensing image generation are bridged.

[0062] In the present invention, the specific method for constructing the foreground object database is not limited, and those skilled in the art can construct it based on experience.

[0063] In a preferred embodiment, each picture in the remote sensing image data set is cropped and segmented, and only images with object content are retained to form a foreground object database.

[0064] More preferably, the object layout annotation is used as a hint, and the objects contained in the remote sensing image dataset are segmented using SAM (Segment Anything Model) to obtain object masks, and all objects are summarized according to object types to complete the construction of the database.

[0065] Preferably, in S3, the object image is input into the visual encoder DINOv2, and the visual features of the foreground object are obtained through the visual encoder DINOv2.

[0066] Preferably, the foreground object features are further projected into the synthesis task space through a trainable multi-layer perceptron model.

[0067] Most existing text-based image models, such as Stable Diffusion, are pre-trained based on general images, which have large domain differences relative to remote sensing images. In the present invention, by setting up a foreground object database and then retrieving images matching the objects, the authenticity of remote sensing images can be effectively improved.

[0068] In S4, a diffusion model is used to generate multiple instances and synthesize remote sensing images.

[0069] Preferably, the diffusion model adopts a diffusion model based on LDM architecture.

[0070] Furthermore, in the diffusion model, an enhanced attention layer is set to enhance the feature authenticity of each instance (i.e., each object).

[0071] Preferably, the object layout description and foreground object features are used as conditions to guide the enhanced attention layer, thereby ensuring that the attributes and position of each instance meet the user-specified requirements.

[0072] The remote sensing image synthesis method in the present invention adopts a divide-and-conquer strategy, decomposing the multi-instance generation task into multiple subtasks, each of which involves the generation of a single instance feature, thereby improving the diversity and controllability of the synthesis results and ensuring that the number, category and spatial distribution of objects in the synthesized image are consistent with the text description.

[0073] According to the present invention, the training process of all network parameters is realized through the existing remote sensing image database through the S1-S4 process, which is the same as the training process of the traditional neural network. After the training is completed, the remote sensing image corresponding to the given structured text can be obtained through the steps S2-S4.

[0074] Various embodiments of the method described above in the present invention can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs, which may be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general programmable processor, which may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0075] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in the present disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document is not limited here.

[0076] Example

[0077] Example 1

[0078] Perform remote sensing image generation, including:

[0079] S1. Construct structured text with information about objects and their spatial distribution;

[0080] S2, establish a mapping between structured text and image layout;

[0081] S3, according to the image layout prediction, obtaining the foreground object image matching the layout prediction from the foreground object database, and encoding to obtain the visual features of the foreground object;

[0082] S4. Based on image layout prediction and structured text description, remote sensing images are synthesized with the guidance of the visual features of foreground objects.

[0083] A remote sensing image database is used to train parameters in all networks through the S1-S4 process. After the training is completed, a remote sensing image corresponding to the given structured text is obtained through steps S2-S4 by giving the given structured text.

[0084] In S1, Figure 2 As shown, the image is divided into 7×7 image blocks, the 3×3 image blocks located at the center of the image are called the "central area", and the remaining image blocks are named according to their relative positions compared to the central area (such as "upper area" and "lower right area").

[0085] There are 5 sentences in each structured text. The first sentence describes the name and specific number of each type of object in order from most to least. The second sentence describes the name and specific number of objects in the central area image block in order from most to least. The third to fifth sentences describe the name, specific number and spatial distribution of objects in other area image blocks in order from most to least.

[0086] Before describing the image, the number of objects is sorted by category, and the three categories with the largest number of objects are selected as the target categories. When describing, only the target categories and their specific numbers are described. When there are less than three target categories, the description sentences are left blank to ensure that the total number of sentences in each structured text is the same.

[0087] In S2, the Transformer-based autoregressive network is used to learn the spatial position relationship between objects in the image by establishing the sequential dependency relationship between objects in the text, thereby realizing the mapping between structured text and image layout. In the process of training the Transformer-based autoregressive network, the object category is first predicted {l i}(i=1,2,…,N, N is the total number of object categories) and the image block number to which the center point belongs {k k}(j=1,2,…,K 2 ), and then regress the specific object box parameters (x, y, w, h) based on this. After the training is completed, the mapping between structured text and image layout is generated as follows Figure 3 As shown in the figure, the yellow dot is the predicted center point of the object, and the boxes of different colors represent the object boxes of different categories of targets. It is not difficult to see that the method proposed in the present invention can accurately predict the center point of the specified category of object based on the text, and further regress to obtain the corresponding object box.

[0088] In S3, the object image is input into the visual encoder DINOv2, and the visual features of the foreground object are obtained through the visual encoder DINOv2.

[0089] In S4, a diffusion model based on the LDM architecture is used to generate multiple instances and synthesize remote sensing images.

[0090] After training is complete, given the structured text:

[0091] Structured Text 1. There are two planes in the center of the image;

[0092] Structured Text 2. There is a highway toll booth in the center of the image;

[0093] Structured Text 3. There is an airport in the center of the image;

[0094] Structured Text 4. There is a dam in the center of the image;

[0095] Structured Text 5. There are two chimneys in the center of the image;

[0096] Structured Text 6. There are two highway service areas in the center of the image and a vehicle at the bottom of the image.

[0097] The mapping between structured text and image layout is as follows Figure 4 As shown, after steps S2-S4, the corresponding remote sensing image is generated as shown in Figure 5 As shown in the figure, it can be seen that the generated remote sensing image objects are reasonably laid out, meet the user's intentions, and accurately reproduce the complex relationships and details in the text.

[0098] The present invention has been described above in conjunction with preferred embodiments, but these embodiments are only exemplary and serve only as an illustration. On this basis, the present invention may be subjected to a variety of substitutions and improvements, all of which fall within the scope of protection of the present invention.

Claims

1. A controllable generation method of remote sensing images based on multimodal information guidance, characterized in that: include: S1. Construct structured text with information about objects and their spatial distribution; S2, establish a mapping between structured text and image layout; S3, according to the image layout prediction, obtaining the foreground object image matching the layout prediction from the foreground object database, and encoding to obtain the visual features of the foreground object; S4. Based on image layout prediction and structured text description, remote sensing images are synthesized with the guidance of the visual features of foreground objects.

2. The method for controllable generation of remote sensing images based on multimodal information guidance according to claim 1, characterized in that: The image is divided into multiple image blocks, the relative positions of the image blocks where the objects are located are described, and structured text with information about the objects and their spatial distribution is obtained.

3. The method for controllable generation of remote sensing images based on multimodal information guidance according to claim 1, characterized in that: The structured text also includes a description of the number of objects in the image block.

4. The method for controllable generation of remote sensing images based on multimodal information guidance according to claim 1, characterized in that: The structured text has a plurality of sentences, including: Describe the name and specific quantity of each type of object in order of quantity; Describe the names and specific quantities of objects in the image block in the central area in order of quantity; The names, specific quantities and spatial distribution of objects in other regional image blocks are described in order of quantity.

5. The method for controllable generation of remote sensing images based on multimodal information guidance according to claim 1, characterized in that: Before describing the image, the number of objects is sorted by category, and several categories with the largest number of objects or categories with a number of objects greater than a threshold are selected as target categories. When describing, only the target categories and their specific numbers are described.

6. The method for controllable generation of remote sensing images based on multimodal information guidance according to claim 1, characterized in that: In S2, an autoregressive network with a Transformer architecture is used to learn the spatial position relationship between objects in the image by establishing sequential dependencies between objects in the text.

7. The method for controllable generation of remote sensing images based on multimodal information guidance according to claim 1, characterized in that: In S3, the foreground object image is input into the visual encoder DINOv2, and the visual features of the foreground object are obtained through the visual encoder DINOv2.

8. The method for controllable generation of remote sensing images based on multimodal information guidance according to claim 1, characterized in that: In S4, a diffusion model is used to generate multiple instances and synthesize remote sensing images.

9. An electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-8.

Images