A controllable generation method of remote sensing images based on multi-modal information guidance

Abstract

The application discloses a kind of based on multi-modal information guide's remote sensing image controllable generation method, comprising: structured text with object and its spatial distribution information is constructed;Establish the mapping between structured text and image layout;According to image layout prediction, obtain the foreground object image matched with layout prediction from foreground object database, encode and obtain foreground object visual features;Based on image layout prediction and structured text description, with foreground object visual features as guide, synthesize remote sensing image.The method disclosed in the application can generate remote sensing images with corresponding semantics according to given condition information, while ensuring the number, category and spatial distribution of objects in the image have diversity and complexity.

Description

TECHNICAL FIELD

[0001] The present application relates to a kind of based on multi-modal information guidance remote sensing image controllable generation method, belong to image synthesis technical field. BACKGROUND

[0002] With the rapid development of earth observation technology and imaging sensor performance, the size and resolution of remote sensing image have been significantly improved. This makes the parameter quantity of target extraction network increase sharply, and further puts forward higher requirements for the size of training data set. However, the acquisition and labeling process of remote sensing data is time-consuming and laborious, involving a large amount of manpower and material resources.

[0003] The text-to-image method based on diffusion model has the ability to generate images based on text description, which can be used for remote sensing data synthesis or labeling in theory, but in fact, its framework structure and training target are mostly designed for natural image (such as face, animal, building, etc.) synthesis task, and are mainly applied to game, painting and photography fields. Compared with natural images, the objects in remote sensing images have the characteristics of large scale difference, less texture features, dense spatial distribution, random object direction, etc., so the data distribution of the two is significantly different, which makes it difficult for the existing text-to-image method based on diffusion model to directly realize the synthesis of remote sensing images.

[0004] In addition, although the existing remote sensing image generation method can guide the synthesis of images with semantic conditions such as text description, these methods still have obvious limitations in controlling the number, category and spatial position of objects in the output image, and it is difficult to synthesize training samples according to specific task requirements or model performance, which greatly reduces the value of synthesized images as training data, and seriously restricts their application in remote sensing visual perception model training.

[0005] Therefore, it is necessary to conduct in-depth research on remote sensing image generation method to solve the above problems. SUMMARY

[0006] In order to overcome the above problems, in-depth research has been conducted, and a kind of remote sensing image controllable generation method based on multi-modal information guidance is proposed, which includes:

[0007] S1, construct a structured text with object and its spatial distribution information;

[0008] S2, establish the mapping between structured text and image layout;

[0009] S3, according to the image layout, obtain the foreground object image matched with the layout prediction from the foreground object database, and encode the obtained foreground object visual features;

[0010] S4, based on image layout prediction and structured text description, guided by foreground object visual features, synthesize remote sensing images.

[0011] In a preferred embodiment, the image is divided into multiple image blocks, and the relative position of the image block where the object is located is described, thereby obtaining a structured text with object and spatial distribution information.

[0012] In a preferred embodiment, the structured text further includes a description of the number of objects in the image block

[0013] In a preferred embodiment, the structured text has multiple sentences, including:

[0014] The name and specific number of each type of object are described in order of quantity;

[0015] The name and specific number of objects in the center region image block are described in order of quantity;

[0016] The name, specific number and spatial distribution of objects in other region image blocks are described in order of quantity.

[0017] In a preferred embodiment, before describing the image, the number of objects is sorted by category, and a number of categories with the largest number of objects or categories with a number of objects greater than a threshold value are selected as target categories, and only the target categories and their specific numbers are described during description.

[0018] In a preferred embodiment, in S2, an autoregressive network with a Transformer architecture is used to learn the spatial position relationship between objects in the image by establishing the sequential dependency relationship between objects in the text.

[0019] In a preferred embodiment, in S3, the foreground object image is input into a visual encoder DINOv2, and the foreground object visual features are obtained through the visual encoder DINOv2.

[0020] In a preferred embodiment, in S4, a diffusion model is used for multi-instance generation to synthesize remote sensing images.

[0021] The application also provides an electronic device, comprising:

[0022] At least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of the above methods.

[0023] The application further provides a computer readable storage medium storing computer instructions, wherein the computer instructions are used for enabling the computer to perform the method according to any one of the preceding embodiments.

[0024] The application has the following beneficial effects:

[0025] (1) The reasonable object layout is generated by taking the description text as a guide, so that the intention of the user is adapted, the burden of the user is reduced, and the automation degree of the image generation process is improved;

[0026] (2) The method has good controllability, can generate remote sensing images with corresponding semantics according to given condition information, and can ensure that the number, category and spatial distribution of objects in the image have diversity and complexity;

[0027] (3) The high-true remote sensing image is effectively generated. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 FIG. 1 is a flowchart of a method for controllable generation of remote sensing images based on multi-modal information guidance according to a preferred embodiment of the application;

[0029] Figure 2 FIG. 4 is a schematic diagram of the process of image division and construction of structured text with object and spatial distribution information in embodiment 1;

[0030] Figure 3 FIG. 5 is a schematic diagram of the process of image layout prediction and remote sensing image synthesis in embodiment 1;

[0031] Figure 4 FIG. 6 is a mapping generation result between structured text and image layout in embodiment 1;

[0032] Figure 5 FIG. 7 is a remote sensing image result generated in embodiment 1. DETAILED DESCRIPTION

[0033] The application will be further described in detail below with reference to the accompanying drawings and embodiments. Through these descriptions, the features and advantages of the application will become more apparent.

[0034] The special word "exemplary" herein means "serving as an example, embodiment or illustration". Any embodiment described herein as "exemplary" is not necessarily to be construed as superior or better than other embodiments. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0035] According to the application, a method for controllable generation of remote sensing images based on multi-modal information guidance is provided, as shown in FIG. 1, which comprises the following steps: Figure 1

[0036] ​S1, construct structured text with object and its spatial distribution information;

[0037] S2, establish mapping between structured text and image layout;

[0038] S3, obtain foreground object image matching layout prediction from foreground object database according to image layout prediction, and encode foreground object visual features;

[0039] S4, synthesize remote sensing image based on image layout prediction and structured text description, guided by foreground object visual features.

[0040] The existing natural language control text is difficult to realize the cross-modal semantic consistency between the text and the image, which has a great negative impact on the controllability of the synthesized image. Analysis shows that the above problems are mainly caused by the fact that the existing control text can only summarize the features such as the category and attributes (such as color, shape, etc.) of the visual targets in the remote sensing image, but it is difficult to clearly describe the number and spatial distribution position of the objects. To solve this problem, in the present application, by constructing structured text with object and its spatial distribution information, the spatial information of the target object can be efficiently introduced into the text, and the expression ability of the text description is improved.

[0041] In a preferred embodiment, the image is divided into multiple image blocks, and the relative position of the object in the image block is described to obtain structured text with object and its spatial distribution information.

[0042] More preferably, the structured text further includes a description of the number of objects in the image block.

[0043] In a preferred embodiment, the structured text has multiple sentences, including:

[0044] The name and specific number of each type of object are described in order of quantity;

[0045] The name and specific number of the object in the center region image block are described in order of quantity;

[0046] The name, specific number and spatial distribution of the object in the other region image block are described in order of quantity.

[0047] Preferably, the number of sentences in each structured text is the same, for example, 5.

[0048] According to the present application, the size of the central region image block can be freely set by those skilled in the art according to actual needs, for example, dividing the image into KxK image blocks, taking K / 2xK / 2 image blocks located in the center of the image as the central region image block, and naming the remaining image blocks according to their relative position compared with the central region, such as "upper region", "lower right region", etc.

[0049] Preferably, before describing the image, the number of objects is sorted by category, and a number of categories with the largest number of objects or categories with the number of objects greater than a threshold value are selected as target categories, and when describing, only the target categories and their specific numbers are described.

[0050] According to the present application, when sorting the number of objects of a specific category, the specific setting value of the number of target categories or the threshold value of the number of objects is not limited, and those skilled in the art can freely set it according to actual needs, for example, selecting the three categories with the largest number of objects as target categories.

[0051] Preferably, when the number of objects is greater than a set threshold value, for example, the number of objects is greater than 5, a unified quantifier is used to refer to the number of objects, and the quantifier can be "many", "Many", etc., which is not limited in the present application.

[0052] Preferably, when describing the image, three sentences are used to describe the three categories with the largest number of objects, respectively, and when the target categories are less than three, the description sentence is left blank to ensure that the total number of sentences of each structured text is the same.

[0053] According to the structured text construction method of the present application, the spatial distribution and relative position information of the main objects in the image can be expressed in the shortest language, the semantic ambiguity is reduced as much as possible without describing the precise spatial coordinates of the objects by words, and the method has the advantages of low computational complexity and high use efficiency.

[0054] In a preferred embodiment, an existing large model can be used to construct the structured text.

[0055] For example, a large language model such as BERT (Bidirectional Encoder Representations from Transformers), LLaMA (Large Language Model Meta AI) is used, the large language model is fine-tuned using a remote sensing image dataset, and then directly output; or a GPT-4 (Generative Pre-trained Transformer-4) or Gemini online multi-modal large model is used, and the structured text is obtained by adjusting the prompt words and conducting multi-round dialogue.

[0056] In S2, preferably, an autoregressive network adopting a Transformer architecture is used to learn the spatial positional relationship between objects in the image by establishing the sequential dependency relationship between objects in the text, so as to realize the mapping between the structured text and the image layout, and obtain the image layout prediction through the mapping.

[0057] In a preferred embodiment, in the process of training the autoregressive network adopting the Transformer architecture, the class {l i}(i = 1, 2, …, N, N is the total number of object classes) and the image block sequence number {k j}(j = 1, 2, …, K 2} of the center point of the object are first predicted, and then the specific object frame parameters (x, y, w, h) are regressed on this basis.

[0058] The above-mentioned training strategy from coarse to fine can reduce the difficulty of cross-modal data mapping modeling from text information to spatial semantics, and improve the efficiency and stability of the regression network model training.

[0059] Further, the method successfully realizes the conversion from the description text to the image layout, and has cross-modal semantic consistency.

[0060] In S3, the foreground object database refers to a database composed of only images of objects themselves.

[0061] By removing the background of the object and only keeping the object itself, the interference of the background information on the subsequent generation process is avoided, and the domain difference of the existing generation model established on the general picture dataset in the remote sensing image generation is bridged.

[0062] In the present application, the specific construction method of the foreground object database is not limited, and a person skilled in the art can construct it according to experience.

[0063] In a preferred embodiment, the foreground object database is formed by cropping and segmenting each picture in the remote sensing image dataset and only keeping the images with objects themselves.

[0064] More preferably, the object layout annotation is used as a prompt, the SAM (Segment Anything Model) is used to segment the objects contained in the remote sensing image dataset, the object mask is obtained, all objects are summarized according to the object categories, and the construction of the database is completed.

[0065] Preferably, in S3, the object image is input into the visual encoder DINOv2, and the foreground object visual features are obtained through the visual encoder DINOv2.

[0066] Preferably, the foreground object features are further projected to a synthesis task space by a trainable multi-layer perception model.

[0067] Existing text-to-image models, such as Stable Diffusion, are mostly pre-trained on general images, which have a large domain difference compared to remote sensing images. In the present application, a foreground object database is set up to retrieve images matching the object, which can effectively improve the realism of remote sensing images.

[0068] In S4, a diffusion model is used to generate multiple instances, and a remote sensing image is synthesized.

[0069] Preferably, the diffusion model uses a diffusion model based on the LDM architecture.

[0070] Further, in the diffusion model, an enhanced attention layer is set up to enhance the feature realism of each instance (i.e., each object).

[0071] Preferably, the object layout description and the foreground object features are used as the condition guidance of the enhanced attention layer, so as to ensure that the attributes and positions of each instance meet the requirements specified by the user.

[0072] In the present application, the remote sensing image synthesis method uses a divide-and-conquer strategy to divide the multiple instance generation task into multiple sub-tasks, each of which involves single instance feature generation, thereby improving the diversity and controllability of the synthesis result and ensuring that the number, category, and spatial distribution of objects in the synthesized image meet the text description.

[0073] According to the present application, the training process of the conventional neural network is the same, and the existing remote sensing image database is used to train all the parameters in the network through the S1-S4 process. After the training is completed, the given structured text is input into the network, and the remote sensing image corresponding to the given structured text can be obtained through steps S2-S4.

[0074] The various embodiments of the method described above in the present application can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on chips (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that can be executed and / or interpreted on programmable systems including at least one programmable processor, which can be a special or general purpose programmable processor, can receive data and instructions from a storage system, at least one input device, and at least one output device, and can transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0075] It should be understood that the various forms of flow shown above can be used to reorder, add or delete steps. For example, the steps described in the present disclosure can be performed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions of the present disclosure can be achieved, which is not limited herein.

[0076] Embodiments

[0077] Embodiment 1

[0078] The remote sensing image generation includes:

[0079] S1, constructing a structured text with objects and their spatial distribution information;

[0080] S2, establishing a mapping between the structured text and the image layout;

[0081] S3, according to the image layout prediction, obtaining the foreground object image matching the layout prediction from the foreground object database, and encoding the foreground object visual features;

[0082] S4, based on the image layout prediction and the structured text description, synthesizing the remote sensing image guided by the foreground object visual features.

[0083] The remote sensing image database is used to realize the training of all network parameters through the S1-S4 process. After the training is completed, the remote sensing image corresponding to the given structured text is obtained through steps S2-S4 by giving the structured text.

[0084] In S1, as shown in Figure 2 , the image is divided into 7x7 image blocks, and the 3x3 image blocks located in the center of the image are called "central region", and the remaining image blocks are named according to their relative position compared with the central region (such as "upper region", "lower right region").

[0085] The number of sentences in each structured text is 5. The first sentence describes the name and specific number of each type of object in order from more to less. The second sentence describes the name and specific number of objects in the central region image block in order from more to less. The third to fifth sentences describe the name, specific number and spatial distribution of objects in other region image blocks in order from more to less.

[0086] Before describing the image, the number of objects is sorted by category, and the three types of objects with the largest number of objects are selected as target classes. When the target class is less than three, the description statement is left blank to ensure that the total number of sentences in each structured text is the same.

[0087] In S2, an autoregressive network with a Transformer architecture is used to learn the spatial position relationship between objects in the image by establishing the sequential dependency relationship between objects in the text, so as to realize the mapping between the structured text and the image layout. During the training of the autoregressive network with the Transformer architecture, the class {l i}(i = 1, 2, …, N, N is the total number of object classes) and the image block sequence number {k k}(j = 1, 2, …, K 2 ) of the center point of the object are first predicted, and then the specific object frame parameters (x, y, w, h) are regressed on this basis. After the training is completed, the mapping generation result between the structured text and the image layout is as shown in Figure 3 , and the yellow points in the figure are the predicted object center points, and the square boxes of different colors represent the object frames of objects of different categories. It can be seen that the method proposed in the application can accurately predict the center points of objects of a specified category according to the text, and further regress the corresponding object frame.

[0088] In S3, the object image is input into a visual encoder DINOv2, and the foreground object visual features are obtained through the visual encoder DINOv2.

[0089] In S4, a diffusion model based on an LDM architecture is used for multi-instance generation to synthesize a remote sensing image.

[0090] After the training is completed, given the structured text:

[0091] Structured text one, there are two airplanes in the center of the image;

[0092] Structured text two, there is a highway toll station in the center of the image;

[0093] Structured text three, there is an airport in the center of the image;

[0094] Structured text four, there is a dam in the center of the image;

[0095] Structured text five, there are two chimneys in the center of the image;

[0096] Structured text six, there are two highway service areas in the center of the image, and there is a vehicle in the lower part of the image.

[0097] The mapping between the structured text and the image layout is as shown in Figure 4 , and the corresponding remote sensing image generated through steps S2-S4 is as shown in Figure 5 . As can be seen from the figure, the object layout of the generated remote sensing image is reasonable, meets the user's intention, and accurately reproduces the complex relationship and details in the text.

[0098] The application has been described above with reference to preferred embodiments. However, these embodiments are merely exemplary and are intended to be illustrative only. Various substitutions and alterations are possible in view of the disclosure of this application without departing from the spirit and scope of the application.

Claims

1. A method for controllable generation of remote sensing images based on multi-modal information guidance, characterized in that, Comprising: S1, constructing a structured text with object and its spatial distribution information; S2, learning the spatial position relationship between objects in the image by establishing the sequential dependence relationship between objects in the text through an autoregressive network adopting a Transformer architecture, establishing the mapping between the structured text and the image layout, and obtaining the image layout prediction; S3, according to the image layout prediction, obtaining the foreground object image matched with the layout prediction from the foreground object database, and encoding the foreground object visual features; S4, based on the image layout prediction and the structured text description, synthesizing the remote sensing image by using a diffusion model, and setting an enhanced attention layer in the diffusion model, and taking the foreground object visual features as the condition guidance of the enhanced attention layer.

2. The method of claim 1, wherein the image is divided into a plurality of image blocks, and the relative positions of the objects in the image blocks are described to obtain the structured text with object and its spatial distribution information.

3. The method of claim 1, wherein the structured text further comprises a description of the number of objects in the image blocks.

4. The method of claim 1, wherein the structured text has a plurality of sentences, comprising: describing the name and specific number of each type of object in order; describing the name and specific number of objects in the center region image block in order; describing the name, specific number and spatial distribution of objects in other region image blocks in order.

5. The method of claim 1, wherein before describing the image, the number of objects is sorted by category, and a plurality of categories with the largest number of objects or categories with a number of objects greater than a threshold are selected as target categories, and only the target categories and their specific numbers are described during the description.

6. The method of claim 1, wherein in S3, the foreground object image is input into a visual encoder DINOv2 to obtain the foreground object visual features through the visual encoder DINOv2.

7. An electronic device, comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6. The computer instructions are used to make the computer execute the method of any one of claims 1-6. ​ ​ ​ ​ 8. A computer readable storage medium having stored thereon computer instructions, wherein, ​

Images