Offshore scene image generation method and device

Abstract

The invention provides a maritime scene image generation method and device. The method comprises the following steps: determining a real image of a maritime scene as a background image of an expected generation target; the target type of the target is generated, a mask image is created at the target position where the target is expected to be generated in the background image, and the mask image is matched with the target type; jointly inputting the mask image and the background image into a diffusion model, and converting image information into low-dimensional potential representation by an encoder of the diffusion model; performing forward diffusion and reverse diffusion on the potential representation in the hidden space, obtaining the state of the potential representation at any moment through noise adding processing in the forward diffusion process, performing denoising processing based on a U-Net network in the reverse diffusion process, and obtaining the potential representation of the target image through noise prediction; a target image of the offshore scene is generated based on the target image potential representation. According to the scheme, virtual-real combined marine target training samples are generated, the quality and diversity of the generated samples are improved, and the data acquisition cost is reduced.

Description

Technical Field

[0001] This application belongs to the technical field of image generation, and particularly relates to a method and device for generating images of marine scenes. Background Art

[0002] In the tasks of target recognition and monitoring in marine scenes, complex sea conditions and weather conditions pose great challenges to the detection and recognition of targets. Due to the existence of the long-tail effect, it is not easy to obtain low signal-to-noise ratio image data of targets in complex marine scenes. At this time, it is usually necessary to generate training samples of marine targets to provide support for the training of subsequent target detection and recognition algorithm models.

[0003] Regarding the problem of generating targets in complex marine scenes, the prior art provides two solutions as follows:

[0004] The first prior art is a scheme for generating training samples of marine targets based on target rendering. Specifically, using 3D rendering technology, by configuring a target model, setting a virtual camera, selecting a remote sensing background and lights, a main scene and a temporary scene are constructed. Among them, the main scene generates realistic color images through rendering, and these images contain the targets and backgrounds that want to be presented in the synthetic image; at the same time, the temporary scene is used to generate a mask, and the mask coordinates the images in the color image to ensure the visual consistency between the target and the background. Through the method of the first scheme, it is possible to effectively create a synthetic image rich in information and visually coordinated.

[0005] The second prior art is a scheme for generating marine target samples based on deep learning. Specifically, a deep learning model is used to learn a large number of marine target samples, capture the internal relationship of the data, and generate targets in complex scenes. Among them, commonly used deep learning models include GAN (Generative Adversarial Networks) and VAE (Variational AutoEncoder), etc. Specifically, GAN generates realistic virtual samples through the adversarial training of a generator and a discriminator; VAE generates virtual samples with certain distribution characteristics through the structure of an encoder and a decoder.

[0006] Although the above two schemes can generate marine scene targets, there are also some deficiencies. For the first scheme, the generation of training samples of marine targets based on target rendering can customize the scene setting and target selection, but it requires 3D models of targets and scenes, and the data acquisition cost is relatively high, and the authenticity of the rendered images is affected by the quality of the 3D models. For the second scheme, the sample generation method based on deep learning also has many limitations: when GAN generates complex scenes, it is often difficult to control the quality and diversity of the generated samples, and the training process is unstable; the samples generated by VAE often lack details and realism and are difficult to meet the requirements of marine target recognition. Summary of the Invention

[0007] The present application provides a method and apparatus for generating maritime scene images, aiming to overcome the above deficiencies of the existing maritime target training sample generation scheme based on target rendering and the maritime target sample generation scheme based on deep learning.

[0008] An embodiment of the present application provides a method for generating maritime scene images, the method including:

[0009] Determine the real image of the maritime scene as the background image of the target to be generated;

[0010] Determine the target type of the target to be generated according to the real image, where the target type at least includes ships and buoys;

[0011] Create a mask image at the target position of the target to be generated in the background image, where the size and shape of the mask image both match the target type;

[0012] Input the mask image and the background image into a diffusion model together. The encoder of the diffusion model converts the image information into a low-dimensional latent representation; perform forward diffusion and reverse diffusion on the latent representation in the latent space. The forward diffusion process obtains the state of the latent representation at any moment through noise addition processing. The reverse diffusion process performs denoising processing based on the U-Net network and obtains the latent representation of the target image corresponding to the target type through noise prediction;

[0013] Generate a target image of the maritime scene based on the latent representation of the target image. The target image is a virtual-real combined image obtained after generating the target in the background image.

[0014] Optionally, the step of obtaining the state of the latent representation at any moment through noise addition processing in the forward diffusion process includes:

[0015] Obtain the initial data distribution of the forward diffusion process;

[0016] Gradually add Gaussian noise to the initial data distribution or the data distribution of the previous moment according to a preset standard deviation and mean algorithm to obtain the posterior probability distribution of the latent representation at the current moment; where the standard deviation is determined by reparameterization, and the mean is jointly determined by reparameterization and the state of the latent representation at the current moment;

[0017] Obtain two standard noises according to a preset two-dimensional standard normal distribution and the posterior probability distribution of the current moment, and merge the two standard noises through a parameter renormalization algorithm to obtain the state of the latent representation at any moment.

[0018] Optionally, the steps of positive diffusion and inverse diffusion of the latent representation in the latent space include:

[0019] Using the conditional encoder CLIP to obtain the semantic information of the target type;

[0020] The inverse diffusion process performs denoising based on the U-Net network, and obtains the latent representation of the target image matching the semantic information through noise prediction.

[0021] Optionally, the steps of positive diffusion and inverse diffusion of the latent representation in the latent space include:

[0022] Performing weighted fusion on the input of the diffusion model through a cross-attention mechanism.

[0023] Optionally, the step of performing weighted fusion on the input of the diffusion model through a cross-attention mechanism includes:

[0024] Calculating attention weights for the current state of the latent representation and the conditional information obtained through the conditional encoder CLIP;

[0025] Normalizing the attention weights and performing weighted fusion of the attention weights with the original input features of the diffusion model.

[0026] Optionally, the step of generating the target image of the maritime scene based on the latent representation of the target image includes:

[0027] Inputting the latent representation of the target image into the decoder of the diffusion model, and the decoder converts the information in the latent space into image information in the pixel space to obtain the target image.

[0028] Optionally, the method further includes:

[0029] Evaluating the image quality of the generated target image;

[0030] When the target image does not meet the set image quality and fidelity, repeat the step of adjusting the parameters of the diffusion model until a qualified target image is generated.

[0031] An embodiment of the present application provides a maritime scene image generation device, and the device includes:

[0032] A background image determination module, configured to determine the real image of the maritime scene as the background image of the desired generation target;

[0033] A target type determination module, configured to determine the target type of the desired generation target according to the real image, and the target type includes at least ships and buoys;

[0034] A mask image generation module, configured to create a mask image at a target position in the background image where a target is expected to be generated, and the size and shape of the mask image both match the target type;

[0035] A diffusion model processing module, configured to jointly input the mask image and the background image into a diffusion model, and an encoder of the diffusion model converts image information into a low-dimensional latent representation; perform forward diffusion and reverse diffusion on the latent representation in the latent space, the forward diffusion process obtains the state of the latent representation at any moment through noise addition processing, the reverse diffusion process performs denoising processing based on a U-Net network, and obtains a target image latent representation corresponding to the target type through noise prediction;

[0036] A target image generation module, configured to generate a target image of a maritime scene based on the target image latent representation, and the target image is a virtual-real combined image obtained after generating the target in the background image.

[0037] Optionally, the diffusion model processing module is specifically configured to:

[0038] Obtain an initial data distribution of the forward diffusion process;

[0039] Gradually add Gaussian noise to the initial data distribution or the data distribution of the previous moment according to a preset standard deviation and mean algorithm to obtain the posterior probability distribution of the latent representation at the current moment; wherein, the standard deviation is determined by reparameterization, and the mean is jointly determined by reparameterization and the state of the latent representation at the current moment;

[0040] According to a preset two-dimensional standard normal distribution and the posterior probability distribution at the current moment, obtain two standard noises, and merge the two standard noises through a parameter renormalization algorithm to obtain the state of the latent representation at any moment.

[0041] Optionally, the diffusion model processing module is specifically configured to:

[0042] Use a conditional encoder CLIP to obtain semantic information of the target type;

[0043] The reverse diffusion process performs denoising processing based on a U-Net network, and obtains a target image latent representation that matches the semantic information through noise prediction.

[0044] Optionally, the diffusion model processing module is further configured to perform weighted fusion on the input of the diffusion model through a cross-attention mechanism.

[0045] Optionally, the diffusion model processing module is specifically configured to:

[0046] Calculate attention weights for the current state of the latent representation and the conditional information obtained through the conditional encoder CLIP;

[0047] Normalize the attention weights and perform weighted fusion of the attention weights with the original input features of the diffusion model.

[0048] Optionally, a target image generation module for:

[0049] Input the latent representation of the target image into the decoder of the diffusion model, and the decoder converts the information in the latent space into image information in the pixel space to obtain the target image.

[0050] Optionally, it further includes an image quality evaluation module for:

[0051] Evaluate the image quality of the generated target image;

[0052] When the target image does not meet the set image quality and fidelity, repeat the step of adjusting the parameters of the diffusion model until a qualified target image is generated.

[0053] An embodiment of the present application provides an electronic device, including: a processor, a memory, and a computer program stored on the memory and executable on the processor. When the computer program is executed by the processor, the above method is implemented.

[0054] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above method is implemented.

[0055] As can be seen from the above solution, first, determine the real image of the sea scene as the background image of the target to be generated; then, determine the target type of the target to be generated according to the real image; then, create a mask image at the target position of the target to be generated in the background image, and the size and shape of the mask image both match the target type; then, input the mask image and the background image into the diffusion model together, and the encoder of the diffusion model converts the image information into a low-dimensional latent representation; perform forward diffusion and reverse diffusion on the latent representation in the latent space. In the forward diffusion process, the state of the latent representation at any moment is obtained through noise addition processing, and in the reverse diffusion process, denoising processing is performed based on the U-Net network, and the latent representation of the target image corresponding to the target type is obtained through noise prediction; finally, generate the target image of the sea scene based on the latent representation of the target image, and the target image is a virtual-real combined image obtained after generating the target in the background image.

[0056] As can be seen from the above, this application superimposes a real scene and a virtual target, generates a virtual-real combined marine target image sample through a diffusion model, and realizes the generation of a virtual-real combined marine target sample in a typical optical remote sensing scene, providing data support for the large sample expansion of remote sensing image data. Moreover, by introducing the semantic information of the target type through the conditional encoder CLIP, the generation quality and diversity of the target image sample are improved; further, through the cross-attention mechanism, the weighted fusion ability of the model for different input features is enhanced, and the fidelity and details of the generated image are improved. In addition, compared with the traditional method for making a dataset based on real remote sensing images, this application can also reduce the data acquisition cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 FIG. is a flowchart of a method for generating a marine scene image provided by this application;

[0058] Figure 2 FIG. is a schematic diagram of the U-Net network in the diffusion model provided by this application;

[0059] Figure 3 FIG. is a schematic diagram of the cross-attention mechanism provided by this application;

[0060] Figure 4 FIG. is a flowchart of a method for generating a marine scene target provided by this application;

[0061] Figure 5 FIG. is a structural diagram of a device for generating a marine scene image provided by this application;

[0062] Figure 6 FIG. is a structural diagram of an electronic device provided by this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are some, but not all, of the embodiments of this application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of this application without creative efforts shall fall within the protection scope of this application.

[0064] To overcome the above deficiencies of the existing marine target training sample generation scheme based on target rendering and the marine target sample generation scheme based on deep learning, this application provides a method and device for generating a marine scene image.

[0065] The method for generating a marine scene image provided by this application will be described below.

[0066] Example 1

[0067] As Figure 1 shown, it is a flowchart of a method for generating a maritime scene image provided by this application. The method may include the following steps:

[0068] Step S101: Determine the real image of the maritime scene as the background image of the target to be generated.

[0069] Specifically, real maritime scene images can be collected, and one or several images can be selected from the collected images as the background image or background image set, so as to be superimposed with the subsequently generated virtual target to obtain the target image.

[0070] Step S102: Determine the target type of the target to be generated according to the real image. The target type at least includes ships and buoys. It should be noted that the ships and buoys listed here are only two specific cases of the target type and do not limit the target type. It can be set according to the specific requirements of the actually generated target.

[0071] Step S103: Create a mask image at the target position of the target to be generated in the background image. The size and shape of the mask image both match the target type.

[0072] It should be noted that the mask image is used for image coordination in color images to ensure visual consistency between the target and the background. In addition, the mask image is also used to automatically generate annotation information of the target, which is very important for training and validating machine learning models.

[0073] Step S104: Input the mask image and the background image into the diffusion model together. The encoder of the diffusion model converts the image information into a low-dimensional latent representation; perform forward diffusion and reverse diffusion on the latent representation in the latent space. The forward diffusion process obtains the state of the latent representation at any moment through noise addition processing. The reverse diffusion process is based on the U-Net network for denoising processing, and the latent representation of the target image corresponding to the target type is obtained through noise prediction.

[0074] In one case, the state of the latent representation at any moment can be obtained in the following manner: obtaining the initial data distribution of the forward diffusion process; gradually adding Gaussian noise to the initial data distribution or the data distribution at the previous moment according to a preset standard deviation and mean algorithm to obtain the posterior probability distribution of the latent representation at the current moment; wherein, the standard deviation is determined by reparameterization, and the mean is jointly determined by reparameterization and the state of the latent representation at the current moment; according to a preset two-dimensional standard normal distribution and the posterior probability distribution at the current moment, two standard noises are obtained, and the two standard noises are combined through a parameter renormalization algorithm to obtain the state of the latent representation at any moment.

[0075] In a preferred manner, a conditional encoder CLIP (Contrastive Language-Image Pretraining) can be used to obtain the semantic information of the target type; the inverse diffusion process is denoised based on a U-Net network, and the target image latent representation matching the semantic information is obtained through noise prediction.

[0076] Furthermore, during the process of forward diffusion and inverse diffusion of the latent representation in the latent space, the input of the diffusion model can be weighted and fused through a cross-attention mechanism to control the generation of the target, so as to realize the generation of a clear and realistic target on a real background. Specifically, the attention weights can be calculated for the current state of the latent representation and the conditional information obtained through the conditional encoder CLIP; the attention weights are normalized, and the attention weights are weighted and fused with the original input features of the diffusion model.

[0077] Step S105: Generate a target image of a maritime scene based on the target image latent representation, and the target image is a virtual-real combined image obtained by generating the target in the background image. Specifically, the target image latent representation can be input into the decoder of the diffusion model, and the decoder converts the information in the latent space into image information in the pixel space, thereby obtaining the target image.

[0078] In order to further improve the generation quality of the target image, the method can also evaluate the image quality of the generated target image; when the target image does not meet the set image quality and fidelity, the step of adjusting the parameters of the diffusion model is repeatedly executed until a qualified target image is generated.

[0079] As can be seen from the above, this application superimposes the real scene and the virtual target, generates a sample image of a sea target combining reality and virtuality through a diffusion model, and realizes the generation of a sample of a sea target combining reality and virtuality in a typical optical remote sensing scene, providing data support for the large-scale expansion of remote sensing image data. Moreover, by introducing the semantic information of the target type through the conditional encoder CLIP, the generation quality and diversity of the target image samples are improved; further, through the cross-attention mechanism, the weighted fusion ability of the model for different input features is enhanced, and the fidelity and details of the generated images are improved. In addition, compared with the traditional method of making a dataset based on real remote sensing images, this application can also reduce the data acquisition cost.

[0080] In addition, the vast majority of current sea target sensing and recognition tasks are based on deep learning methods, and deep learning methods are all data-driven and need to be trained on a large number of datasets. However, the traditional method of making a dataset is to directly obtain real remote sensing images and manually annotate them, which is costly and time-consuming. Moreover, it is difficult to collect sample images in complex scenes, which limits the application scope of deep learning models. The adaptability and recognition accuracy of the deep learning detection and recognition model for complex sea scenes are improved.

[0081] Next, taking the ship target as the target type, the sea scene image generation method provided by this application will be described. Please refer to Figures 2 to 4 , Figure 2 is a schematic diagram of the U-Net network in the diffusion model provided by this application, Figure 3 is a schematic diagram of the cross-attention mechanism provided by this application, Figure 4 is a flowchart of the sea scene target generation method provided by this application. Among them, Figure 4 shows the process of the complex sea scene target generation method, and details the complete process from inputting the real sea background image to outputting the target image containing the virtual target. The specific steps are as follows:

[0082] Step (1): Input the parameters of the diffusion model.

[0083] Specifically, load the parameters of the diffusion model. As Figure 4 shown, select a real sea background image (i.e., the background image), and select the position (i.e., the target position) where it is desired to generate a specified target (i.e., a ship) on the background image. Then, form a mask image at this position, and input the mask image and the background image x s together into the encoder ε of the diffusion model. After being processed by the encoder of the diffusion model, the image information in the pixel space is compressed into low-dimensional information in the latent space.

[0084]

[0085] Among them, the main function of the encoder of the diffusion model is to convert the image information x in the pixel space into a low-dimensional latent representation z in the latent space, and this latent representation z is the input of the diffusion process in the diffusion model.

[0086] Step (2): Perform forward diffusion and reverse diffusion on the latent representation z of the image in the latent space.

[0087] In one case, in the reverse diffusion process, the target type information is incorporated in the following way: First, select the target type to be generated, further mine the semantic information from the target type through the conditional encoder CLIP, and input this semantic information into the diffusion model. Further, as Figure 3 shown, the reverse diffusion process is guided through the cross-attention mechanism, thereby controlling the generation of specific targets and achieving the generation of clear and realistic targets on a real background.

[0088] Specifically, the derivation processes of the forward diffusion and reverse diffusion performed in the latent space are as follows:

[0089] (1) In the forward diffusion process, we first have an initial data distribution x 0 ~q(x), and then we gradually add Gaussian noise to this data distribution step by step, adding it once at each moment. It should be noted that both the standard deviation and the mean of the Gaussian noise are related to the pre-set value β t ; among them, the standard deviation is completely determined by β t , while the mean is jointly determined by β t and the state x t at the current moment, and the forward process of the diffusion model is a Markov process.

[0090] Assume that the image distribution at the initial moment is equivalent to the true distribution of the data. Then, taking an image from the training set can be regarded as a sampling from the true distribution of the data. We use x 0 to represent the original image, and x T to represent the result obtained after the original image has gone through T time steps and added T times of noise. Then, the forward process of the diffusion model can be represented by q(x t |x t-1 ), that is, at each step of the forward process, x t-1 is obtained by adding Gaussian noise to the image x t . The mean of the added Gaussian noise is and the variance is ∑ t = β t I. Then,

[0091] the noise addition process can be represented as:

[0092]

[0093] Since the diffusion process is a Markov process, therefore, from x 0 to x T the posterior probability distribution can be expressed as:

[0094]

[0095] To obtain an actionable closed-form sampling at any time step, the reparameterization trick is introduced. First, a two-dimensional standard normal distribution variable is introduced, and then through the reparameterization β t , from x t-1 to obtain x t :

[0096]

[0097] Similarly, we have:

[0098]

[0099] Combining the above expressions (3) and (4), we get:

[0100]

[0101] Then define α t = 1 - β t , then we have β t = 1 - α t , so the above equation can be written as:

[0102]

[0103] Since the two standard noises shown in expressions (5) and (6) satisfy Therefore, we have:

[0104]

[0105] Combining the two noises shown in expressions (5) and (6), through parameter renormalization, it becomes a form containing only one random variable:

[0106]

[0107] And so on, using the parameter renormalization trick, we can directly obtain the state z t at any time:

[0108]

[0109] From expression (7), we only need to know the initial state z 0 of the image and set the fixed value sequence in advance Sampling a standard noise ∈ from the standard normal distribution allows for directly calculating the state at time t in one step.

[0110] (2) Next, consider the reverse diffusion process, i.e., q(z t-1 |x t ). In the diffusion model, the state x at time t t is known, and the denoising process of this process is actually learning the probability distribution function of x t based on x t-1 , which can be achieved by constructing a neural network (i.e., the U-Net network). The reverse diffusion process can be expressed as p θ (x t-1 |x t ), where θ are the network model parameters. Then,

[0111] The model p θ can be expressed as:

[0112]

[0113] According to the Markov chain, the expression of the reverse diffusion process can be obtained:

[0114]

[0115] To predict the probability distribution of the noise, we introduce the posterior diffusion conditional probability:

[0116]

[0117] Considering the posterior diffusion conditional probability, using Bayes' formula and substituting the relationship between x 0 , x t derived above, we can obtain:

[0118]

[0119] (3) In the reverse diffusion process, the diffusion model uses a U-Net convolutional neural network (abbreviated as the U-Net network) to predict the noise. Please refer to Figure 2 and Figure 4 . During the denoising process, the input to the U-Net is the current time t vector and the current state x t, the output is the prediction of the current moment noise. Before inputting into the U-Net, the current moment t needs to be mapped to a vector through the TimeEmbedding module. The TimeEmbedding module is a technique that converts the time step t into an embedding vector of a specified dimension, and is usually used in deep learning models, especially when dealing with time series data. TimeEmbedding obtains vectors by sampling sine and cosine functions, and these vectors are determined by parameters such as time step, vector dimension, and period. In this application, the TimeEmbedding module plays a role similar to positional encoding, telling the U-Net which time step of noise prediction is being performed currently.

[0120] After predicting the noise ∈, we use the current state x t Subtract the noise to obtain the state x of the previous moment t-1 . The same denoising step is continuously performed. After we complete the T denoising processes, we can obtain a clear new image x 0 , Figure 2 Intuitively demonstrates this process.

[0121] (4) Cross-attention fusion, establishing connections between different modules through the attention mechanism to promote information exchange and integration. In the cross-attention mechanism, the sequences to be fused must have the same dimension. As Figure 3 shown, a query vector Q is generated at each position of one sequence as input, and a key vector K and a value vector V are generated at each position of the other sequence. Then, a dot product operation is performed using the vector Q and the vector K, and the attention weights are obtained through the Softmax function. Multiplying the attention weights by the vector V can obtain the fused output sequence.

[0122] In this stage, we calculate the attention weights for the current state in the latent space and the conditional information obtained through the conditional encoder CLIP, then normalize the attention weights using the Softmax function, and then apply the calculated attention weights to the original input features to perform weighted fusion of different input features. Finally, the feature representation after weighted fusion is used as the output of the cross-attention mechanism and passed to the subsequent classification task.

[0123] Specifically, let Q = X 1 W Q , K = X 2 W K , V = X 2 W V , then the calculation formula for the cross-attention score is as follows:

[0124]

[0125] Among them, X 1 , X 2 respectively represent the input sequence 1 and sequence 2, is the learned projection matrix, and d k is the dimension of the key-value set (which is also the dimension of the query set).

[0126] Step (3): Generate the result image.

[0127] Specifically, the generated result latent representation obtained after the inverse diffusion process can be input into the decoder. The decoder converts the information in the latent space into image information in the pixel space, thereby obtaining the result image, that is, generating an image of a virtual ship target in the real sea background.

[0128] It should be noted that for the method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions involved are not necessarily required by this application.

[0129] Next, the sea scene image generation device provided by this application will be described.

[0130] Example 2

[0131] As Figure 5 shown, it is a structural diagram of the sea scene image generation device provided by this application. The device includes: a background image determination module 210, a target type determination module 220, a mask image generation module 230, a diffusion model processing module 240, and a target image generation module 250.

[0132] Among them, the background image determination module 210 is used to determine the real image of the sea scene as the background image of the target to be generated;

[0133] The target type determination module 220 is used to determine the target type of the target to be generated according to the real image, and the target type at least includes ships and buoys;

[0134] The mask image generation module 230 is used to create a mask image at the target position of the target to be generated in the background image, and the size and shape of the mask image both match the target type;

[0135] The diffusion model processing module 240 is configured to jointly input the mask image and the background image into a diffusion model. The encoder of the diffusion model converts the image information into a low-dimensional latent representation. In the latent space, forward diffusion and reverse diffusion are performed on the latent representation. In the forward diffusion process, noise addition processing is performed to obtain the state of the latent representation at any moment. The reverse diffusion process is based on a U-Net network for denoising processing, and the target image latent representation corresponding to the target type is obtained through noise prediction.

[0136] The target image generation module 250 is configured to generate a target image of a maritime scene based on the target image latent representation. The target image is a virtual-real combined image obtained by generating the target in the background image.

[0137] In one case, the diffusion model processing module 240 is specifically configured to obtain the initial data distribution of the forward diffusion process; gradually add Gaussian noise to the initial data distribution or the data distribution at the previous moment according to a preset standard deviation and mean algorithm to obtain the posterior probability distribution of the latent representation at the current moment. Wherein, the standard deviation is determined by reparameterization, and the mean is jointly determined by reparameterization and the state of the latent representation at the current moment; according to a preset two-dimensional standard normal distribution and the posterior probability distribution at the current moment, two standard noises are obtained, and the two standard noises are combined through a parameter renormalization algorithm to obtain the state of the latent representation at any moment.

[0138] In another case, the diffusion model processing module 240 is specifically configured to use the conditional encoder CLIP to obtain the semantic information of the target type; the reverse diffusion process is based on a U-Net network for denoising processing, and the target image latent representation matching the semantic information is obtained through noise prediction.

[0139] Further, the diffusion model processing module 240 is further configured to perform weighted fusion on the input of the diffusion model through a cross-attention mechanism. Specifically, the diffusion model processing module 240 is configured to calculate the attention weight for the current state of the latent representation and the conditional information obtained through the conditional encoder CLIP; perform normalization processing on the attention weight, and perform weighted fusion on the attention weight and the original input features of the diffusion model.

[0140] In one case, the target image generation module 250 is configured to input the target image latent representation into the decoder of the diffusion model. The decoder converts the information in the latent space into image information in the pixel space to obtain a target image.

[0141] Preferably, the device further includes an image quality assessment module for assessing the image quality of the generated target image; when the target image does not meet the set image quality and fidelity, the step of adjusting the parameters of the diffusion model is repeatedly executed until a qualified target image is generated.

[0142] As can be seen from the above, this application superimposes the real scene and the virtual target, and generates a virtual-real combined marine target image sample through a diffusion model. In a typical optical remote sensing scene, it realizes the generation of virtual-real combined marine target samples, providing data support for the large-sample expansion of remote sensing image data. Moreover, by introducing the semantic information of the target type through the conditional encoder CLIP, the generation quality and diversity of the target image samples are improved; further, through the cross-attention mechanism, the weighted fusion ability of the model for different input features is enhanced, improving the fidelity and details of the generated images. In addition, compared with the traditional method for making a dataset based on real remote sensing images, this application can also reduce the data acquisition cost.

[0143] In addition, the vast majority of current marine target sensing and recognition tasks are based on deep learning methods, and deep learning methods are all data-driven and need to be trained on a large number of datasets. However, the traditional method for making a dataset is to directly obtain real remote sensing images and manually annotate them, which is costly, time-consuming and laborious. Moreover, it is difficult to collect sample images in complex scenes, restricting the application scope of deep learning models. It improves the adaptability and recognition accuracy of the deep learning detection and recognition model for complex marine scenes.

[0144] It should be noted that for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for the relevant parts, please refer to the description in the method embodiment section.

[0145] Example 3

[0146] To solve the above problems, the present invention provides an electronic device, such as Figure 6 shown, including a memory 310, a processor 320, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the above method applied to the electronic device is implemented.

[0147] It should be noted that the electronic device may be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The electronic device may include, but is not limited to, a processor 320 and a memory 310. Those skilled in the art can understand, Figure 6The following are merely examples of electronic devices and do not constitute limitations on electronic devices. They may include more or fewer components than shown in the figures, or combine certain components, or have different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc.

[0148] The so-called processor 320 may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0149] The memory 310 may be an internal storage unit of the electronic device, such as the hard disk or memory of the electronic device. The memory 310 may also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on the electronic device. Further, the memory 310 may also include both the internal storage unit and the external storage device of the electronic device. The memory 310 is used to store the computer program and other programs and data required by the electronic device. The memory 310 may also be used to temporarily store data that has been output or will be output.

[0150] Example 4

[0151] An embodiment of the present invention also provides a computer-readable storage medium. The computer-readable storage medium may be the computer-readable storage medium included in the memory in the above embodiment; or it may be a computer-readable storage medium that exists separately and is not assembled into the commuting broadcast control device. The computer-readable storage medium stores one or more computer programs, and when the programs are executed by a processor, the above-described method is implemented.

[0152] When the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-described embodiment methods of the present invention, it can also be completed by a computer program instructing relevant hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-described various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0153] Although the embodiments disclosed in the present invention are as above, the above content is only an embodiment adopted for the convenience of understanding the present invention and is not intended to limit the present invention. Any person skilled in the art within the technical field to which the present invention pertains can make any modifications and changes in the form of implementation and details without departing from the spirit and scope disclosed by the present invention. However, the protection scope of the present invention shall still be subject to the scope defined by the appended claims.

[0154] The above is the preferred embodiment of the present application. It should be noted that for those of ordinary skill in the art in this technical field, without departing from the principle of the present application, several improvements and refinements can still be made, and these improvements and refinements should also be regarded as within the protection scope of the present application.

Claims

1. A method for generating a marine scene image, characterized in that: The method comprises: Determine the real image of the sea scene as the background image of the target to be generated; Determining the target type of the target to be generated according to the real image, wherein the target type includes at least a ship and a buoy; Creating a mask image at a target position where a target is expected to be generated in the background image, wherein the size and shape of the mask image match the target type; The mask image and the background image are inputted into a diffusion model together, and the encoder of the diffusion model converts the image information into a low-dimensional potential representation; the potential representation is forward diffused and reverse diffused in the latent space, the forward diffusion process obtains the state of the potential representation at any time by adding noise, and the reverse diffusion process performs denoising based on a U-Net network, and obtains the target image potential representation corresponding to the target type by noise prediction; A target image of a marine scene is generated based on the target image potential representation, wherein the target image is a virtual-real combined image obtained after the target is generated in the background image.

2. The method for generating a marine scene image according to claim 1, characterized in that: The step of obtaining the state of the potential representation at any time by adding noise in the forward diffusion process comprises: Obtain the initial data distribution of the positive diffusion process; According to a preset standard deviation and mean algorithm, Gaussian noise is gradually added to the initial data distribution or the data distribution at the previous moment to obtain the posterior probability distribution of the potential representation at the current moment; wherein the standard deviation is determined by the weight parameter, and the mean is determined by the weight parameter and the state of the potential representation at the current moment; According to the preset two-dimensional standard normal distribution and the posterior probability distribution at the current moment, two standard noises are obtained, and the two standard noises are combined through a parameter renormalization algorithm to obtain the state of the potential representation at any moment.

3. The method for generating a marine scene image according to claim 1, characterized in that: The step of performing forward diffusion and reverse diffusion on the potential representation in the latent space comprises: Using a conditional encoder CLIP to obtain semantic information of the target type; The inverse diffusion process performs denoising based on a U-Net network, and obtains a potential representation of the target image that matches the semantic information through noise prediction.

4. The method for generating a marine scene image according to claim 3, characterized in that: The step of performing forward diffusion and reverse diffusion on the potential representation in the latent space comprises: The inputs of the diffusion model are weightedly fused through a cross-attention mechanism.

5. The method for generating a marine scene image according to claim 4, characterized in that: The step of weighted fusion of the input of the diffusion model by means of a cross attention mechanism comprises: Calculating attention weights for the current state of the latent representation and the conditional information obtained by the conditional encoder CLIP; The attention weights are normalized and weightedly fused with the original input features of the diffusion model.

6. The method for generating marine scene images according to claim 1, characterized in that: The step of generating a target image of a marine scene based on the target image potential representation comprises: The target image potential representation is input into the decoder of the diffusion model, and the decoder converts the information in the latent space into image information in the pixel space to obtain the target image.

7. The method for generating a marine scene image according to claim 1, characterized in that: Also includes: Performing image quality assessment on the generated target image; When the target image does not reach the set image quality and fidelity, the step of adjusting the parameters of the diffusion model is repeated until a qualified target image is generated.

8. A device for generating a marine scene image, characterized in that: The device comprises: A background image determination module, used to determine a real image of a marine scene as a background image of a desired generated target; A target type determination module, used to determine the target type of the target to be generated according to the real image, wherein the target type includes at least a ship and a buoy; A mask image generation module, used to create a mask image at a target position where a target is expected to be generated in the background image, wherein the size and shape of the mask image are matched with the target type; A diffusion model processing module, used for inputting the mask image and the background image into a diffusion model, wherein the encoder of the diffusion model converts the image information into a low-dimensional potential representation; performing positive diffusion and reverse diffusion on the potential representation in a latent space, wherein the positive diffusion process obtains the state of the potential representation at any time by adding noise, and the reverse diffusion process performs denoising based on a U-Net network, and obtains the target image potential representation corresponding to the target type by noise prediction; The target image generation module is used to generate a target image of a marine scene based on the potential representation of the target image, wherein the target image is a virtual-real combined image obtained after the target is generated in the background image.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method according to any one of claims 1 to 7 when executed by the processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images