Offshore unknown scene image synthesis method and device based on diffusion model

Abstract

The embodiment of the invention provides a maritime unknown scene image synthesis method and device based on a diffusion model. The method comprises the following steps: adjusting the posture of a foreground target, and generating target images with different postures by inputting a single foreground target image; generating background images under different environment and weather conditions by inputting a single background image and semantic information; and performing image fusion by using a neural network by calculating and adjusting the size and proportion of the foreground image in the background image. By means of the target and image synthesis method, image synthesis of a new scene is achieved, simulation of an unknown scene can be rapidly achieved, the unknown scene serves as a training and verification data set of intelligent algorithms such as detection and recognition of a marine target and environment, the problem that marine task scene data is deficient can be solved with low cost and high efficiency, and the target and image synthesis efficiency is improved. A cultivation environment is provided for intelligent training and ability verification of the offshore unmanned system, and iterative upgrading of the intelligent level of the offshore unmanned system is accelerated.

Description

Technical Field

[0001] The present application relates to the technical field of generating virtual environments at sea, and more particularly to a method and device for synthesizing images of unknown scenes at sea for training and testing intelligent perception algorithms, and more particularly to a method and device for synthesizing images of unknown scenes at sea based on a diffusion model. Background Art

[0002] The rapid development of unmanned intelligent technology at sea has put forward huge new demands on the ability to simulate the complex marine environment in a refined manner, which is of great significance to accelerating the development and application of unmanned systems at sea. There are problems such as low efficiency, high cost, and long cycle in the actual ship test at sea and the test methods based on real data. Virtual test and training of unmanned systems in the mission scenario at sea has become one of the important solutions. However, the construction theory and method of the virtual environment of traditional unmanned systems at sea are relatively poor in scene modeling accuracy and sensor simulation accuracy, resulting in the virtual environment can only reflect the real situation in a relatively qualitative and logical way, and the training and testing efficiency of intelligent algorithms or intelligent systems has decreased.

[0003] Although the industry has carried out some research work on marine environment simulation, the sensor simulation accuracy is generally low, which makes it difficult to meet the training and verification needs of intelligent perception algorithms. Wang W et al. developed an unmanned boat intelligent driving simulator based on the Unity platform, and carried out high-precision modeling for environmental weather, waves, visible light, radar and other modules. Although the modeling accuracy has been improved, it is difficult to cover up the problems of simple models and environmental distortion. Xiao G et al. also developed an unmanned boat simulation platform based on Unity, and used CFD simulation technology to accurately model water flow, wind field and other environments, but ignored key issues such as scene construction and sensor modeling.

[0004] Traditional image fusion methods simply place the foreground image on the background image; resulting in the synthesized image having several defects in the following aspects.

[0005] Position distortion - Foreground objects appear in unreasonable locations within the background image; for example, a buoy in the sky or a ship on the seafloor.

[0006] Sizing distortion – The sizes of foreground and background images are imbalanced; for example, a ship appears smaller than a buoy or a seagull appears smaller.

[0007] Pose distortion - Inconsistency in shooting angles between foreground and background images causes foreground objects to appear strange.

[0008] Fusion artifacts - Differences in lighting and texture between foreground and background images lead to strange fusion effects with discontinuous boundaries.

[0009] The emergence of the above problems leads to a large domain gap between synthetic data and real captured data; therefore, models trained on synthetic data perform poorly in real environments. Summary of the invention

[0010] In order to solve the above problems, an embodiment of the present invention provides a method and device for synthesizing images of unknown marine scenes based on a diffusion model.

[0011] An embodiment of the present invention provides a method for synthesizing an image of an unknown marine scene based on a diffusion model, the method comprising:

[0012] Adjust the pose of the foreground target and generate target images with different poses by inputting a single foreground target image;

[0013] Generate background images under different environments and weather conditions by inputting a single background image and semantic information;

[0014] The image fusion is performed using a neural network by computationally adjusting the size and ratio of the foreground image in the background image.

[0015] In some embodiments, generating target images of different poses by inputting a single foreground target image includes:

[0016] Taking in a single foreground image, we fine-tune the latent diffusion model to generate new views using an optimized objective function in the latent space of a variational autoencoder.

[0017] In some embodiments, generating background images under different environments and weather conditions by inputting a single background image and semantic information includes:

[0018] Receive a single background image and a semantic condition;

[0019] Apply the latent diffusion model to recover the target image from the latent space of Gaussian noise image;

[0020] Encode the text prompt.

[0021] In some embodiments, the image fusion using a neural network includes:

[0022] Given a set of foreground images and background image β = {b1, b2, ..., b n}, synthesize and generate a new marine image dataset D = {d1, d2, ..., d0}, the synthesis process is as follows:

[0023] d i =M(f i , b i );

[0024] Among them, M is the image fusion model.

[0025] In some embodiments, the synthesis process specifically includes:

[0026] d i =M ψ (A θ (f i ), G φ (b i ))

[0027] Among them, the foreground adjustment module A parameterized by θ θ , the background generation module G parameterized by φ φ and the image fusion module M parameterized by ψ ψ .

[0028] An embodiment of the present invention provides a device for synthesizing images of unknown marine scenes based on a diffusion model, the device comprising:

[0029] The foreground adjustment module is used to adjust the posture of the foreground target and generate target images with different postures by inputting a single foreground target image;

[0030] The background generation module is used to synthesize images of ocean scenes under specific semantic content, and to generate background images under different environmental and weather conditions by inputting a single background image and semantic information;

[0031] The image fusion module is used to calculate the size and ratio of the adjusted foreground image in the background image and perform image fusion through a neural network.

[0032] An embodiment of the present invention provides an electronic device, including a processor and a memory storing program instructions, wherein the processor is configured to execute a method as described in any of the above embodiments when running the program instructions.

[0033] An embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. The computer program is designed to execute the method described in any of the above embodiments.

[0034] The beneficial effects of the above-mentioned embodiments of the present invention include: the synthesis effect of the maritime position image based on the diffusion generative model is more realistic, and the degree of intelligence and efficiency are higher. By using the target and image synthesis method of the present invention for test scenes that have not been carried out in the real physical world at sea, the image synthesis of the new scene is realized, and the simulation of unknown scenes can be quickly realized. It can be used as a training and verification data set for intelligent algorithms such as detection and identification of maritime targets and environments, which can solve the problem of lack of maritime mission scene data at low cost and high efficiency, provide a nurturing environment for intelligent training and capability verification of maritime unmanned systems, and accelerate the iterative upgrade of the intelligence level of maritime unmanned systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The drawings illustrate generally, by way of example and not limitation, various embodiments discussed herein.

[0036] Figure 1 A schematic flow chart of a method for synthesizing an image of an unknown marine scene based on a diffusion model is provided for an embodiment of the present invention;

[0037] Figure 2 Schematic diagram for determining parameter y. DETAILED DESCRIPTION

[0038] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0039] In the embodiments of the present application, it should be noted that, unless otherwise specified and limited, the term "connection" should be understood in a broad sense. For example, it can be an electrical connection or a connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0040] It should be noted that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects, and do not represent a specific order for the objects. It is understandable that the specific order or sequence of "first\second\third" can be interchanged where permitted. It should be understood that the objects distinguished by "first\second\third" can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0041] This method is based on the diffusion generative model and realizes the synthesis of marine environment background images and typical target images. It has the function of multi-target synthesis under the marine background in unknown scenes.

[0042] Given a set of foreground images and background image β = {b1, b2, ..., b n}, the proposed method aims to generate a new marine image dataset D = {d1, d2, ..., d0} through image synthesis. The foreground image contains key marine targets such as ships and buoys. The background image includes various marine environments and climates. The image synthesis process can be expressed as the following equation:

[0043] d i =M(f i , b i ) (1)

[0044] Where M is the image fusion model.

[0045] This method proposes a new 3D perception image synthesis method. The overview of this method is as follows Figure 1 As shown in Figure 2, the method for synthesizing unknown marine scene images based on the diffusion model consists of three modules, including a foreground adjustment module A parameterized by θ. θ , the background generation module G parameterized by φ φ and the image fusion module M parameterized by ψ ψ . The process can be expressed by the following equation:

[0046] d i =M ψ (A θ (f i ), G φ (b i )) (2)

[0047] Among them, the foreground adjustment module A θ Aims to adjust the pose of the foreground target. Generate target images with different poses by inputting a single foreground target image. Background generation module G φ The role of is to generate background images for various environments and weather conditions. By inputting a single background image and semantic information, background images can be generated under different conditions. Finally, the image fusion module M ψ By calculating and adjusting the size and ratio of the foreground image in the background image, a neural network is used for image fusion. The above three modules all use a neural network model to enhance the ability of image adjustment and fusion, making the synthesized image close to the real image.

[0048] 1. Foreground adjustment module

[0049] The function of the foreground adjustment module is to In the case of in and represents the relative camera rotation and translation. The process can be expressed as the following equation:

[0050]

[0051] To be estimated A new view f that is perceptually similar to the real but unobserved one R，T For this type of problem, by fine-tuning the latent diffusion model ∈ θ To optimize the following objective function in the variational autoencoder latent space:

[0052]

[0053] where z is the latent representation of the input image encoded by the model ε, t is the time step, ∈ is the Gaussian noise image, and c(f, R, T) is the view-conditional representation. θ Fine-tuning is performed on publicly available paired images and their relative camera extrinsics. This process builds zero-shot capabilities into the model so that object classes that have never appeared in the fine-tuning set can also be synthesized. To further improve visual quality, like Wonder3D, this application adopts a cross-domain diffusion model to generate multi-view normal maps and corresponding color images as a Markov chain during fine-tuning.

[0054] 2. Background generation module

[0055] The purpose of the background generation module is to generate a background image Under the condition of The process can be expressed as the following equation:

[0056]

[0057] The semantic condition S can be represented by any environmental factor, such as weather, climate, lighting, etc. Referring to ControlNet, the latent diffusion model ε is applied φ (z t , t, c(b, S)) from the Gaussian noise image z t The target image is recovered from the latent space of , where c(b, S) is the latent feature space of the input image and semantic conditions, and S is the representation of the semantic conditions, i.e., the prompt. The text prompt is encoded using the CLIP model. Finally, the background generation module is fine-tuned under the following objective function:

[0058]

[0059] In this way, specific semantic conditions can be iteratively recovered from Gaussian noise images. This process is flexible, so various target background images can be synthesized.

[0060] 3.3D affine image fusion module

[0061] After getting the adjusted foreground image and background image After that, the image fusion module is applied to combine them to synthesize the final image. The corresponding expression is expressed by formula (2). The traditional image synthesis method simply puts the foreground image on the background image, which has the following two problems. The first is the distortion of the target shape. This is because the spatial mapping relationship is not considered in the image fusion process, resulting in the inconsistency between the proportion and position of the target at a specific distance in the background image and the real world. The second is the distortion of the edge of the fused image. This is because the edge gradient transformation between the foreground and background images is not considered, resulting in the discontinuity of the fused image at the edge position. Therefore, this application proposes a three-dimensional affine image fusion module to solve these problems.

[0062] The 3D affine image fusion module combines the affine transformation in 3D space with the diffusion model. In the structure, the role of the 3D affine transformation is to provide a text embedding consistent with the real world for the diffusion model, so that it can better describe the shape of the foreground image in the background image, while the continuity of the edge can be achieved through the spatial generalization ability of the diffusion model. Specifically, for the diffusion part, the AnyDoor model architecture with the following training objectives is adopted:

[0063]

[0064] Where c(b, c, P) represents the text embedding of the position details P = {x, y, s} of the encoded foreground image f on the background b. Where x, y, s are conditional parameters, representing the horizontal coordinate, vertical coordinate and scale in the pixel coordinate system of the background image, respectively. Traditional methods do not consider these parameters or set them to random values, resulting in distortion of the foreground object in the background image. Here, the present application proposes a method for determining these conditional parameters based on the 3D affine principle.

[0065] First, the determination of x affects the horizontal position of the foreground target in the background image. This parameter is essentially determined by the azimuth of the target in the observer's coordinate system. In the actual data capture process, the positions of the targets appearing on the screen are randomly distributed on both sides of the central axis. Therefore, a Gaussian distribution is used to simulate the value of x, where the mean is the center of the background image width and the variance is one-sixth of the background image width. In this way, it is ensured that the target appears on the screen and has the highest probability of appearing on the central axis, which is consistent with the real pattern.

[0066] Secondly, the determination of y affects the vertical position of the foreground target in the background image. This parameter is essentially determined by the height of the sea level and the distance between the target and the observer. When shooting at real sea, the distance can usually be obtained through a device called a navigation radar. The relationship reflected on the background image is as follows Figure 2 As shown, the values ​​near the bottom of the image are smaller, and the values ​​near the sea level are larger. The relationship between distance and y is not linear, so this application uses an exponential function to simulate the distance of y values. In this way, this application associates the vertical axis of the target in the background image with its distance, making it consistent with the distribution of real data.

[0067] Finally, the determination of parameter s affects the size of the target in the background image. This parameter can be determined by the perspective principle. Assuming that the real height of the target H, the focal length of the camera f, and the measurement of its pixel height h are known, the relationship between the distance to the target d can be calculated using the following equation:

[0068]

[0069] where d new is the new distance at which the target appears in the synthesized image. In this way, the present application associates the scale with the target distance, ensuring that the target scale of the synthesized image is consistent with the target scale of the real image.

[0070] Finally, the parameters generated by the above method are combined with the diffusion model and used as conditional input. This ensures that the diffusion model obtains the correct size of the target during image generation, while the diffusion model itself ensures the quality of image generation.

[0071] An embodiment of the present invention provides an electronic device, including a processor and a memory storing program instructions, wherein the processor is configured to execute a method as described in any of the above embodiments when running the program instructions.

[0072] An embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. The computer program is designed to execute a method as described in any of the above embodiments.

[0073] The technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0074] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for synthesizing images of unknown marine scenes based on a diffusion model, characterized in that: The method comprises: Adjust the pose of the foreground target and generate target images with different poses by inputting a single foreground target image; Generate background images under different environments and weather conditions by inputting a single background image and semantic information; The image fusion is performed using a neural network by computationally adjusting the size and ratio of the foreground image in the background image.

2. The method according to claim 1, characterized in that The method of generating target images of different postures by inputting a single foreground target image comprises: Taking in a single foreground image, we fine-tune the latent diffusion model to generate new views using an optimized objective function in the latent space of a variational autoencoder.

3. The method according to claim 1, characterized in that The method of generating background images under different environments and weather conditions by inputting a single background image and semantic information comprises: Receive a single background image and a semantic condition; Apply the latent diffusion model to recover the target image from the latent space of Gaussian noise image; Encode the text prompt.

4. The method according to claim 1, characterized in that The method of using a neural network to perform image fusion includes: Given a set of foreground images and background image β = {b1, b2, ..., b n }, synthesize and generate a new marine image dataset D = {d1, d2, ..., d0}, the synthesis process is as follows: d i =M(f i ,b i ); Among them, M is the image fusion model.

5. The method according to claim 4, characterized in that The synthesis process specifically includes: d i =M ψ (A θ (f i ),G φ (b i )) Among them, the foreground adjustment module A parameterized by θ θ , the background generation module G parameterized by φ φ and the image fusion module M parameterized by ψ ψ .

6. A device for synthesizing images of unknown marine scenes based on a diffusion model, characterized in that: The device comprises: The foreground adjustment module is used to adjust the posture of the foreground target and generate target images with different postures by inputting a single foreground target image; The background generation module is used to synthesize images of ocean scenes under specific semantic content, and to generate background images under different environmental and weather conditions by inputting a single background image and semantic information; The image fusion module is used to calculate the size and ratio of the adjusted foreground image in the background image and perform image fusion through a neural network.

7. An electronic device comprising a processor and a memory storing program instructions, characterized in that: The processor is configured to perform the method according to any one of claims 1 to 5 when executing the program instructions.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is designed to execute the method according to any one of claims 1 to 5 .

Images