Infrared ship image controllable generation method for offshore scene

Abstract

The invention provides an infrared ship image controllable generation method for a marine scene. The method comprises the following steps: preparing an infrared marine ship image data set; a foreground and background separation controllable ship image generation framework is constructed, and the framework adopts a denoising diffusion model architecture and comprises three stages of foreground target generation, background target generation and foreground and background target fusion; setting model training constraints; respectively training the sub-diffusion models in the generation framework; and inputting a to-be-generated ship target, a position, a camera visual angle and a sea condition into the generation framework, and outputting a target ship generation result in the infrared marine scene meeting the control information requirement by the model, thereby completing ship image generation of the marine infrared scene. By applying the technical scheme of the invention, the technical problem that the existing diffusion model control method cannot realize controllable generation under the conditions of small marine ship target scale, fuzzy ship attribute information and different sea conditions and other interferences can be solved.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular relates to a controllable generation method of infrared ship images for marine scenes. Background Art

[0002] In recent years, the field of visual synthesis has made significant progress. Especially since the rise of deep learning technology, generative paradigms such as generative adversarial networks (GAN) and denoising diffusion models have been proposed one after another, greatly promoting related research. These technologies make the generated results more diverse and more natural and realistic, and realize the transformation of synthetic data from random generation to controllable generation. At present, this field has made significant progress in synthesis quality, attribute controllability, cross-modal synthesis and driving (such as text to image translation), data accumulation, and the extension of pre-trained generative models to specific downstream tasks.

[0003] However, due to the small scale of marine target data in infrared scenes and the low contrast and details of infrared images, the image quality is low, which has largely hindered the development of visual synthesis in this field and related research is relatively scarce. In addition, the unclear texture of small targets and complex sea conditions further increase the difficulty of generating specific types of ships. At present, mainstream diffusion models, such as the Stable Diffusion (SD) model, are mainly pre-trained in the visible light range. Direct fine-tuning based on these models may produce color artifacts in some cases and is difficult to meet the requirements in terms of target control.

[0004] Recently, some studies have begun to focus on how to control diffusion models to achieve a balance between controllability and diversity. The DreamBooth method proposed by Nataniel Ruiz et al. inputs a small number of sample images of specific objects or styles together with predefined category labels into the model, and uses reconstruction loss to enable the model to learn how to generate images with these features. During the fine-tuning process, DreamBooth also adjusts the weights of the model so that it can retain the ability to generate diverse images while generating images with specific personalized features. Another idea is to use weight separation to learn specific attribute features while maintaining the diversity and prior knowledge of generation. ControlNet by Lvmin Zhang et al. is a representative of this idea. ControlNet first locks the original model (locked copy) so that it does not participate in training, and then copies a trainable network model (trainable copy) for training. By inputting control conditions (such as hint embedding or other image latent space feature information) into the network, it passes through zero convolution and trainable copy modules in sequence, and the final output result is added to the result of the locked copy module to obtain the final generated result. This approach allows for more extensive control over the generated results based on multiple types of images, such as edge contours, segmentation maps, depth maps, pose maps, etc.

[0005] Most of the research on background control focuses on target transfer, that is, transferring the target object to a specific background to achieve background control. Methods such as Paint-by-Example and ObjectStitch use the guidance of the image text encoder to embed the reference object into the image, but only retain the semantic similarity with the inserted object. AnyDoor uses the self-supervised representation of the reference object and its high-frequency mapping as a condition to enhance the preservation of the object identity, and supplements the identity features of the self-supervised representation with the detailed features of the high-frequency mapping. This design allows for a variety of local changes (such as lighting, direction, posture, etc.), supports the effective fusion of objects with different environments, and thus improves the fidelity of the images generated by AnyDoor. However, in infrared scenes, the color information prior contained in the pre-trained weights of these models may interfere with the generation of infrared images. At the same time, some physical priors from visible light (such as shadows) may be misleading in infrared scene generation. In addition, there is still a lack of mature solutions for controllable generation methods for small targets and special backgrounds, and this challenge urgently needs in-depth research. Summary of the invention

[0006] The present invention aims to solve at least one of the technical problems existing in the prior art.

[0007] The present invention provides a controllable generation method of infrared ship images for marine scenes, the method comprising:

[0008] Prepare infrared marine ship image dataset;

[0009] A controllable ship image generation framework with foreground and background separation is constructed. The framework adopts a denoising diffusion model architecture and includes three stages: foreground target generation, background target generation, and foreground and background target fusion.

[0010] Set model training constraints;

[0011] Train the sub-diffusion models in the generation framework separately;

[0012] The ship target, position, camera perspective and sea conditions to be generated are input into the generation framework, and the model output is the target ship generation result in the infrared marine scene that meets the control information requirements, completing the ship image generation of the marine infrared scene.

[0013] By applying the technical solution of the present invention, a controllable generation method of infrared ship images for marine scenes is provided. The method constructs a complete controllable diffusion model generation framework in which the foreground and background can be controlled separately. After training, the framework can generate target ships that clearly show the characteristics of unique species, and realize high-realistic foreground and background fusion and position perception fusion. The method of the present invention particularly emphasizes the precise binding of small-scale targets in categories and attributes (such as direction and other information), while focusing on how to accurately simulate the modal characteristics in infrared scenes. Compared with the prior art, the technical solution of the present invention can solve the technical problem that the existing diffusion model control method cannot achieve controllable generation when the marine ship target is small in scale, the ship attribute information is fuzzy, and under different sea conditions and other interferences. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The included drawings are used to provide a further understanding of the embodiments of the present invention, which constitute a part of the specification, are used to illustrate the embodiments of the present invention, and together with the text description, explain the principles of the present invention. Obviously, the drawings in the following description are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0015] Figure 1 A schematic flow chart of a method for controllable generation of infrared ship images for marine scenes provided according to a specific embodiment of the present invention is shown;

[0016] Figure 2 A schematic diagram of a controllable ship image generation framework with foreground and background separation provided according to a specific embodiment of the present invention is shown;

[0017] Figure 3 A synthetic effect diagram of infrared images of ships in a marine scene provided according to a specific embodiment of the present invention is shown. DETAILED DESCRIPTION

[0018] It should be noted that, in the absence of conflict, the embodiments in this application and the features in the embodiments can be combined with each other. The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0020] Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values ​​of the parts and steps set forth in these embodiments do not limit the scope of the present invention. At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the accompanying drawings are not drawn according to actual proportional relationships. The technology, methods and equipment known to those of ordinary skill in the relevant art may not be discussed in detail, but in appropriate cases, the technology, methods and equipment should be considered as a part of the specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values.

[0021] like Figure 1 As shown, according to a specific embodiment of the present invention, a controllable generation method of infrared ship images for marine scenes is provided, and the method includes:

[0022] Prepare infrared marine ship image dataset;

[0023] A controllable ship image generation framework with foreground and background separation is constructed. The framework adopts a denoising diffusion model architecture and includes three stages: foreground target generation, background target generation, and foreground and background target fusion.

[0024] Set model training constraints;

[0025] Train the sub-diffusion models in the generation framework separately;

[0026] The ship target, position, camera perspective and sea conditions to be generated are input into the generation framework, and the model output is the target ship generation result in the infrared marine scene that meets the control information requirements, completing the ship image generation of the marine infrared scene.

[0027] By applying this configuration, a controllable generation method of infrared ship images for marine scenes is provided. The method constructs a complete controllable diffusion model generation framework in which the foreground and background can be controlled separately. After training, the framework can generate target ships that clearly show the characteristics of specific species and realize high-realistic foreground and background fusion and position perception fusion. The method of the present invention particularly emphasizes the precise binding of small-scale targets in categories and attributes (such as direction and other information), and pays attention to how to accurately simulate the modal characteristics in infrared scenes.

[0028] Furthermore, in the present invention, an infrared marine ship image dataset is firstly prepared.

[0029] The present invention selects a suitable infrared ship image data set, completes image preprocessing, and divides the data into a training set, a validation set, and a test set to ensure the effectiveness of model training.

[0030] Furthermore, in the present invention, after acquiring the infrared marine ship image dataset, a foreground-background separation controllable ship image generation framework is constructed.

[0031] In the present invention, the framework adopts a denoising diffusion model architecture, which includes four sub-type diffusion models. Among them, the first and second sub-type diffusion models focus on learning to generate precise categories and attribute bindings for small-scale targets, and adopt a pipeline architecture. Specifically, the first sub-type diffusion model is responsible for learning the contour information of the target, and passes the generated image mask as control information to the second sub-type diffusion model; the second sub-type diffusion model learns the internal texture details of the target on this basis, and finally generates an infrared foreground target with obvious category attributes. In addition, the third sub-type diffusion model generates an infrared sea surface background of a specified viewing angle and sea conditions through the joint control of text information and sketch information. The fourth sub-type diffusion model realizes the high-realistic fusion and position-controllable fusion of the foreground and background in the infrared scene.

[0032] like Figure 2 As shown in the figure, the foreground-background separation controllable ship image generation framework includes three stages: foreground target generation, background target generation and foreground-background target fusion.

[0033] (1) Foreground target generation stage

[0034] The present invention adopts a novel progressive generation strategy, which sequentially learns the contour and texture of the target, and passes the contour learned in the first stage as control information to the second stage to form constraints and guidance. This method effectively improves the problem of poor generation quality caused by low resolution of small-scale targets and fuzzy attribute information. The specific structural process of the foreground target generation stage is as follows:

[0035] 1) Text embedding of perspective and direction information: The required perspective and angle information is embedded through a frozen text encoder to ensure the accuracy of the generated effect.

[0036] 2) Control sketch image embedding: Based on the text, a control sketch can be added and embedded through a frozen image encoder to enhance the control information and reduce the learning difficulty.

[0037] 3) ControlNet outline generation structure: ControlNet locks all parameters in the UNet of the stable diffusion model and clones them into a trainable copy, which is trained as a control network. It is mainly responsible for extracting control sketch information in the first stage and guiding the model to generate corresponding results.

[0038] 4) Mask image embedding: The target mask generated by the first stage of training is used as the control information of the second stage and embedded using a frozen image encoder to reduce the difficulty of learning.

[0039] 5) ControlNet texture generation structure: Similar to the ControlNet contour generation structure in the third step, it is responsible for extracting the information of the mask map in the second stage and guiding the model to perform the final foreground generation.

[0040] Since the UNet model of the stable diffusion model receives latent features (dimension is 64×64) instead of the original image, the image-based input conditions need to be converted to a 64×64 feature space to match the convolution size in the model. Instead of processing the image into a latent space, a lightweight convolutional network ε is directly used here to extract conditional information features:

[0041] c f =ε(c i ),

[0042] The purpose of this step is to input the condition c i Encoded as control feature map c f .

[0043] ControlNet replicates the encoder and intermediate mapping modules in the original stable diffusion model UNet as a controllable encoding network, which is then connected to the decoder of UNet through a jump connection. Its forward process can be expressed as:

[0044]

[0045] Among them, m and y c They represent the deep features of the image encoded by the neural network and the features after transformation, respectively, and c represents the additional conditions provided. z(;) represents zero convolution operation, Represents the mapping operation of the intermediate module of the neural network. Parameter Θ z1 , Θ z2 and θ c Represent the parameters of the first zero-convolutional layer, the second zero-convolutional layer, and the trainable copy, respectively.

[0046] Furthermore, the main training object of ControlNet is the weights of the replicated replica network, while the UNet in the stable diffusion model remains frozen. The loss function used in training is the loss function L used to fit the noise in the original stable diffusion model. simple :

[0047]

[0048] Among them, c t and c f They are text input and control feature map, respectively, which means that the stable diffusion model in ControlNet essentially receives dual-conditional input; represents the computational expectation, ∈ represents the latent space noise, ∈ θ represents the latent space noise calculated by the stable diffusion model, z t It represents the latent space feature when the diffusion time step is t, and t represents the diffusion time step.

[0049] (2) Background target generation stage

[0050] Based on the existing stable diffusion model, the present invention converts the image channel in the post-processing stage to avoid the generation of visible light ghosting. This method effectively improves the visible light prior interference that may be encountered when migrating from visible light to infrared scenes. The specific structural process of the background target generation stage is as follows:

[0051] 1) Text embedding of background description information: The perspective and sea condition information required for generation are embedded through a frozen text encoder to ensure that the generated image meets the expected conditions.

[0052] 2) Stable Diffusion Model Structure: The stable diffusion model is a Wensheng graph model based on the latent diffusion model (Latent Diffusion Models, LDMs for short). Compared with the Denoising Diffusion Probabilistic Models (DDPMs for short), LDMs introduces an autoencoder, which allows the diffusion process to be carried out in the latent space, improving the efficiency of image generation. At the same time, a conditional mechanism is added so that image generation can be controlled by other modal data, and conditional generation control is achieved through the attention mechanism.

[0053] Furthermore, since the diffusion model of LDM acts on the latent space, an encoder ε is required b Embed the original image x into the latent space and get z:

[0054] z=ε b (x),

[0055] Therefore, the objective function of the stable diffusion model in latent space can be simplified as:

[0056]

[0057] This is still a loss calculation for the noisy prediction result, where E ε(x),∈～N(0,1),t Represents computational expectation.

[0058] In addition, LDM transforms the diffusion model into a more flexible conditional image generator by using a cross-attention mechanism in the UNet model. This is done by introducing a new encoder τ for conditional encoding. θ To map condition c to Then Mapped to the middle layer of UNet through the cross attention layer. Therefore, the objective function of LDM under conditional control becomes:

[0059]

[0060] Encoder τ θ and UNet∈ θ Joint training is performed through the above formula.

[0061] 3) Image conversion module: The weighted synthesis method is used to simulate the infrared sensing effect through different weight combinations. Since infrared cameras respond differently to light of different wavelengths, and the red, green, and blue channels of RGB images correspond to different spectral ranges, the performance of objects in the infrared band can be approximately simulated through reasonable weight distribution.

[0062] (3) Foreground and background target fusion stage

[0063] This paper adopts a novel AnyDoor target fusion diffusion model, which aims to characterize the foreground target as "ID-related" and "detail-related" features, and then interactively recombines with the generated background environment features. The specific structural process of the foreground and background target fusion stage is as follows:

[0064] 1) High-frequency map extraction: Generate the required viewpoint and angle information and embed it through a frozen text-side encoder to obtain the high-frequency features of the background.

[0065] Specifically, in order to obtain a high-frequency map, a high-pass filter is used to extract the high-frequency area, and then the Hadamard product is used to extract the RGB color. In addition, an erosion mask is added to filter out the information near the outer contour of the target object. It can be expressed as:

[0066]

[0067] Among them, K h and K v Respectively represent the horizontal and vertical Sobel kernels, used as high-pass filters; M erode represents the erosion mask; I represents the background image, I h Represents a high frequency graph.

[0068] 2) Customized position embedding: By pasting the high-frequency map of the foreground object to the specified position in the background, position-controlled object fusion is achieved. This process ensures the natural fusion of the foreground and background and enhances the spatial consistency of the generated image.

[0069] 3) ID Extractor: A pre-trained autoregressive model is used to extract the identity features of the foreground object, thus providing important semantic information for the generation process.

[0070] 4) Detail Extractor: A ControlNet-style UNet encoder is used to generate a series of detail maps with hierarchical resolutions. These detail maps can enhance the visual representation of foreground objects, making them more realistic and vivid.

[0071] 5) Image generation module: Image generation is performed by injecting the information obtained from the above feature extraction and detail extraction into a stable diffusion model. In this process, the encoder module remains frozen, while the decoder module and other unfrozen parts are jointly trained to achieve more refined image generation.

[0072] Furthermore, the objective function of AnyDoor model optimization adopts a variant of LDM optimization objective function, and the prediction target selects the original data. The formula is written as:

[0073]

[0074] Where x is the real image, t is the diffusion time step, and the pre-trained UNet is denoted as x θ , which is derived from the initial latent space noise Start denoising and use text embedding c as a condition to generate new latent space features Among them, z t represents the latent space feature when the diffusion time step is t, α t and σ t are denoising hyperparameters.

[0075] Furthermore, in the present invention, after completing the construction of the foreground-background separation controllable ship image generation framework, model training constraints are set.

[0076] The present invention adopts the simplified optimization objective based on prediction noise ∈ proposed by Denoising Diffusion Probabilistic Models (DDPM) and its variants as the loss function to constrain the model training process.

[0077] Furthermore, in the present invention, after setting the model training constraints, the sub-diffusion models in the generation framework are trained separately.

[0078] In the present invention, a back propagation algorithm is used to iteratively update and optimize network parameters until the model loss tends to converge, thereby ensuring that each model can effectively learn its target features.

[0079] Furthermore, in the present invention, after completing the training of the sub-diffusion model in the generation framework, the ship target, position, camera perspective and sea conditions to be generated are input into the generation framework, and the model output is the target ship generation result in the infrared marine scene that meets the control information requirements, thereby completing the ship image generation of the marine infrared scene.

[0080] In summary, in view of the shortcomings of the existing diffusion model control methods, the present invention proposes a method for controllable generation of foreground and background of infrared ship images for marine scenes. The framework of this method is based on the serial and parallel generation strategy of the diffusion model, which successfully solves the difficulties of existing algorithms in comprehensive control of foreground and background, and achieves good results in infrared scenes. For the generation of foreground targets, the present invention introduces a novel progressive generation strategy, which effectively improves the generation quality problems caused by low resolution of small-scale targets and blurred attribute information. This strategy significantly improves the clarity and accuracy of the generated results. On the basis of ensuring high-quality generation of foreground and background, the fusion method of the present invention realizes high-realistic foreground and background fusion, and has the function of position perception, meeting the requirements of position control.

[0081] After training, the framework of the present invention can generate target ships that clearly show the characteristics of specific types, and realize the fusion of foreground and background with high realism and position perception. The method particularly emphasizes the precise binding of small-scale targets in categories and attributes (such as direction information), and pays attention to how to accurately simulate the modal characteristics in infrared scenes.

[0082] In order to have a further understanding of the present invention, the controllable generation method of infrared ship images for marine scenes of the present invention is described in detail below in conjunction with specific embodiments.

[0083] This embodiment provides a controllable generation method of infrared ship images for marine scenes, which specifically includes the following steps.

[0084] 1. Preparation of infrared marine ship dataset, including dataset selection, data preprocessing and dataset division.

[0085] 1.1 Dataset A high-quality infrared marine ship dataset is selected. The dataset includes more than 8,400 labeled infrared data collected in different scenes. The image resolutions are:

[0086] 384*288, 640*512, 1280*1024, and seven types of ship targets in the images were labeled.

[0087] 1.2 Dataset preprocessing includes using liner, bulk carrier, sailboat, canoe, containership,

[0088] Fishing boat is used as the label of cruise ship, bulk carrier, kayak, canoe, container ship and fishing boat respectively. The objects are marked with rectangular boxes. The upper left corner of the image is the origin of coordinates [0,0]. The position of the rectangular box is recorded in the form of [x1, y1, x2, y2]. x1 represents the upper left corner horizontal coordinate of the rectangular box, y1 represents the upper left corner vertical coordinate of the rectangular box, x2 represents the lower right corner horizontal coordinate of the rectangular box, and y2 represents the lower right corner vertical coordinate of the rectangular box. All label information is saved in the form of XML files.

[0089] 1.3 The dataset is divided according to the standards given by the publisher. A total of 8402 images are used for training.

[0090] 1000 unlabeled images were used for testing.

[0091] 2. Design the foreground and background to generate a fusion framework. Figure 2As shown, the foreground and background generation fusion framework is based on four diffusion models, including a novel progressive generation strategy to achieve fine control of small-scale foreground targets and high-realistic position perception fusion of foreground and background. The specific model design has been discussed in the previous content and will not be repeated here. In this embodiment, background target generation, foreground target generation, and foreground and background target fusion use Stable Diffusion v1.4, StableDiffusion v1.5, and StableDiffusion v2.1 as basic generators respectively.

[0092] 3. Design model training constraints. For conditional diffusion models such as Stable Diffusion, classifier-free guidance (CFG) is used in the reasoning stage:

[0093] ∈ pred =∈ uc +β cfg (∈ c -∈ uc ),

[0094] Among them, β cfg is a fixed scaling factor, which is set to 0.5 in this embodiment, ∈ uc and ∈ c are the noise predicted by the unconditional diffusion model (text is empty) and the conditional diffusion model. Since the foreground target generation and background target fusion stage models are both dual-conditional diffusion models, a balanced mode is adopted to add all conditional information to ∈ uc and ∈ c On, that is and ∈ u =∈ θ (z t ,t,c t ,c f ), Represents empty, i.e. no text input, thereby constraining the model to treat text and images as equally important.

[0095] 4. Training model. Back propagation algorithm is used to update and optimize the network parameter weights until the model loss converges. In this embodiment, the training and evaluation of the four sub-diffusion models are completed on the PyTorch platform. The model is trained on a single NVIDIA L20 GPU (48GB) with a batch size of 1. The diffusion model is optimized using the Adam optimizer with an initial learning rate of 1×10 -5 After training, the learning rate is adjusted to 1×10 -6 Fine-tune the model.

[0096] 4.1 For the foreground target generation stage: 8 epochs are required for model training to fitting, of which 2 epochs are required for fine-tuning.

[0097] 4.2 For the background target generation stage: 5 epochs are required for model training to fitting, of which 2 epochs are required for fine-tuning.

[0098] 4.3 For the foreground and background target fusion stage: Since it is migrated from visible light to infrared mode, only 2 rounds of fine-tuning are required.

[0099] 5. After the model training is completed, the infrared image of the ship at sea is generated. The control conditions are input into the training generation model, and the model output is the generation result. The present invention can effectively complete the task of generating infrared images of ships in marine scenes, and the attribute binding performance is good on different ship types, such as Figure 3 shown.

[0100] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A controllable generation method of infrared ship images for marine scenes, characterized in that: The controllable generation method of infrared ship images for marine scenes includes: Prepare infrared marine ship image dataset; A controllable ship image generation framework with foreground and background separation is constructed. The framework adopts a denoising diffusion model architecture and includes three stages: foreground target generation, background target generation, and foreground and background target fusion. Set model training constraints; Train the sub-diffusion models in the generation framework separately; The ship target, position, camera perspective and sea conditions to be generated are input into the generation framework, and the model output is the target ship generation result in the infrared marine scene that meets the control information requirements, completing the ship image generation of the marine infrared scene.

2. The controllable generation method of infrared ship images for marine scenes according to claim 1 is characterized in that: The foreground-background separation controllable ship image generation framework includes four sub-diffusion models. The first and second sub-diffusion models focus on learning to generate precise categories and attribute bindings of small-scale targets, and adopt a pipeline architecture. The first sub-diffusion model is responsible for learning the contour information of the target and passing the generated image mask to the second sub-diffusion model as control information; the second sub-diffusion model learns the internal texture details of the target on this basis, and finally generates an infrared foreground target with obvious category attributes; the third sub-diffusion model generates an infrared sea surface background of a specified perspective and sea conditions through the joint control of text information and sketch information; the fourth sub-diffusion model realizes the high-realistic fusion and position-controllable fusion of the foreground and background in the infrared scene.

3. The controllable generation method of infrared ship images for marine scenes according to claim 1 is characterized in that: The specific structural process of the foreground target generation stage is as follows: 1) Text embedding of perspective and direction information: The required perspective and angle information is embedded through a frozen text encoder; 2) Control sketch image embedding: Based on the text, a control sketch is added and embedded through a frozen image encoder; 3) ControlNet outline generation structure: ControlNet locks all parameters in the UNet of the stable diffusion model and clones them into a trainable copy, which is trained as a control network. It is responsible for extracting control sketch information in the first stage and guiding the model to generate corresponding results; 4) Mask image embedding: The target mask generated by the first stage training is used as the control information of the second stage and embedded using the frozen image encoder; 5) ControlNet texture generation structure: Similar to the ControlNet contour generation structure in the third step, it is responsible for extracting the information of the mask map in the second stage and guiding the model to perform the final foreground generation.

4. The controllable generation method of infrared ship images for marine scenes according to claim 3 is characterized in that: ControlNet replicates the encoder and intermediate mapping modules in the original stable diffusion model UNet as a controllable encoding network, which is then connected to the decoder of UNet through a jump connection. Its forward process is expressed as: Among them, m and y c They represent the deep features of the image encoded by the neural network and the transformed features, respectively, and c represents the additional conditions provided; represents zero convolution operation, Represents the mapping operation of the intermediate module of the neural network; parameter Θ z1 , Θ z2 and θ c Represent the parameters of the first zero-convolutional layer, the second zero-convolutional layer, and the trainable copy, respectively.

5. The controllable generation method of infrared ship images for marine scenes according to claim 4 is characterized in that: The loss function L used in ControlNet training simple for: Among them, c t and c f They are text input and control feature maps; represents the computational expectation, ∈ represents the latent space noise, ∈ θ represents the latent space noise calculated by the stable diffusion model, z t It represents the latent space feature when the diffusion time step is t, and t represents the diffusion time step.

6. The controllable generation method of infrared ship images for marine scenes according to claim 1 is characterized in that: The specific structural process of the background target generation stage is as follows: 1) Background description information text embedding: The perspective and sea condition information required for generation are embedded through a frozen text encoder to ensure that the generated image meets the expected conditions; 2) Stable Diffusion Model Structure: The stable diffusion model is a text-based graph model based on the latent space diffusion model. The latent space diffusion model introduces an autoencoder so that the diffusion process is carried out in the latent space. At the same time, a conditional mechanism is added so that image generation is controlled by other modal data, and conditional generation control is achieved through the attention mechanism. 3) Image conversion module: It uses weighted synthesis method to simulate infrared sensing effect through different weight combinations.

7. The controllable generation method of infrared ship images for marine scenes according to claim 6 is characterized in that: In the stable diffusion model structure, the objective function L of the latent space diffusion model is LDM for: Among them, E ε(x),∈～N(0,1),t represents the expected value, τ θ (c) is the condition c through the encoder τ θ After the mapping.

8. The controllable generation method of infrared ship images for marine scenes according to claim 7 is characterized in that: The AnyDoor target fusion diffusion model is used in the foreground and background target fusion stage. The specific structure process is as follows: 1) High-frequency image extraction: Generate the required perspective and angle information and embed it through the frozen text-side encoder to obtain the high-frequency features of the background; 2) Customized position embedding: By pasting the high-frequency image of the foreground target to the specified position in the background, the position-controllable target fusion is achieved; 3) ID Extractor: Uses a pre-trained autoregressive model to extract the identity features of the foreground object; 4) Detail extractor: uses a ControlNet-style UNet encoder to generate a series of detail maps with hierarchical resolution; 5) Image generation module: Image generation is performed by injecting the information obtained from feature extraction and detail extraction into a stable diffusion model.

9. The controllable generation method of infrared ship images for marine scenes according to claim 1 is characterized in that: The objective function L optimized by the AnyDoor model is: in, represents the computational expectation, x is the real image, and the pre-trained UNet is denoted as x θ , α t and σ t are denoising hyperparameters.

10. The controllable generation method of infrared ship images for marine scenes according to claim 1 is characterized in that: The simplified optimization objective based on prediction noise proposed by the denoising probability diffusion model and its variants are used as the loss function to constrain the model training process.

Images