A method for generating a complex background SAR ship image

Abstract

The application discloses a complex background SAR ship image generation method, comprising the following steps: acquiring labels at least including ships and oceans; constructing a complex background SAR ship image model for generating a complex background SAR ship image; training the complex background SAR ship image model to obtain a trained complex background SAR ship image model; and inputting the labels at least including ships and oceans into the trained complex background SAR ship image model to generate a complex background SAR ship image based on the input labels. The application can effectively solve the problems of lacking fine modeling of complex scene elements, being unable to reconstruct high-frequency features such as shore texture and terrain changes, and serious target omission and mispositioning in dense maritime scenes in the current SAR ship generation method, and simultaneously effectively saves the time of researchers through a label-first and image-second generation mode, thereby providing technical support for SAR image processing algorithms.

Description

Technical Field

[0001] This invention belongs to the field of SAR image processing, and particularly relates to a method for generating SAR images of ships with complex backgrounds. Background Technology

[0002] As an active radar system, Synthetic Aperture Radar (SAR) can detect and image targets by synthesizing target echo signals received at different locations. This unique imaging mechanism enables it to penetrate adverse weather conditions such as clouds and fog, making it promising for both civilian and military applications. However, this also places higher demands on SAR image processing technology. Data-driven convolutional neural networks are an emerging image processing model in recent years, but their direct application to SAR image processing still faces significant challenges. This is mainly due to the difficulty of acquiring SAR images and the time-consuming and labor-intensive annotation process, resulting in a limited size of available datasets. Considering these issues, we should vigorously develop SAR ship image generation models to promote the advancement of image processing algorithms.

[0003] In recent years, researchers have applied different models to SAR image generation and achieved some results. However, early synthetic aperture radar (SAR) generation methods focused on generating single ship targets at the image center, neglecting the fine modeling of complex scene elements. Subsequent background generation models were still limited by coarse background representations, failing to reconstruct high-frequency features such as coastal textures and terrain variations. While layout-guided diffusion models achieved breakthroughs in background detail fidelity, they introduced uncontrollable randomness during ship synthesis, resulting in severe target omissions and misalignments in dense maritime scenes. This invention decomposes SAR image generation into three independent paths: ship generation, background generation, and ship-background fusion. On one hand, a encoder-decoder structure with a semantic injection module (SIM) is used to generate ship targets. On the other hand, a diffusion model with mask-aware cross-attention (MaCA) and multi-scale aggregation (MSA) modules is used to generate the background. Finally, a ship-background fusion machine with a spectral attention module (SAM) is used to generate high-quality SAR images. This invention provides new ideas and technical support for SAR image generation methods and has significant research value and significance. Summary of the Invention

[0004] To address the aforementioned problems, the technical solution adopted by this invention is: a method for generating SAR ship images with complex backgrounds, comprising the following steps: Obtain tags that include at least ships and the ocean; Construct a SAR ship image model with complex backgrounds to generate SAR ship images with complex backgrounds; Based on manually labeled tags and real images, a SAR ship image model with complex backgrounds is trained to obtain a well-trained SAR ship image model with complex backgrounds. By inputting labels that include at least ships and ocean into a trained SAR ship image model with complex backgrounds, SAR ship images with complex backgrounds can be generated based on the input labels.

[0005] Furthermore, the complex background SAR ship image model includes: Decoupling module: Used to decouple based on the input labels to obtain an image containing only the ship target, an image containing only the background, and a mask image of the ship target; Diffusion model: Based on the background-only image output by the decoupling module, a background image is generated; Encoder-decoder structure: Based on the image containing only the ship target and the mask image of the ship target output by the decoupling module, the ship target image is generated; Ship background fusion unit: Based on the background image output by the diffusion model and the ship target image output by the encoder-decoder structure, it realizes the fusion of SAR images.

[0006] Furthermore, the label uses a rotated bounding box label, and the format of the rotated bounding box label is defined using OpenCV's definition method: including the x-coordinate of the center point, the y-coordinate of the center point, the length of the first side rotated counterclockwise around the horizontal axis, the length of the adjacent sides of the first side, and the angle between the horizontal axis and the first side rotated counterclockwise, the angle range of which is specified. ; The process of decoupling based on the input labels to obtain an image containing only the ship target, an image containing only the background, and a mask image of the ship target is as follows: The decoupling of ships and background targets in existing SAR images is performed using rotated bounding box labels. The specific process is as follows: The ship mask image is generated based on the rotated bounding box label, and the background mask image is obtained by inverting the ship mask image. The image containing only ships is obtained by multiplying the ship mask image and the original input image pixel by pixel. This operation retains the pixel values ​​of the ships while setting the pixel values ​​of the background to zero, thereby achieving effective separation between the ships and the background. The image containing only the background is obtained by multiplying the background mask and the original image pixel by pixel. For the resulting blank areas, the surrounding background pixel information is used to fill and repair them.

[0007] Furthermore, the diffusion model includes: diffusion-generated structural entities; The diffusion-generated structure includes: Forward diffusion network: used to gradually add random noise to an image that contains only background based on the output of the decoupling module, eventually obtaining a noisy image; Backward denoising network: Based on the full-noise image output by the forward diffusion network, it predicts the noise added at each step through training and removes it from the full-noise image to restore the input image.

[0008] Furthermore, the backward denoising network includes: Mask-aware cross-attention module: used to acquire mask-related information and generate the determined position of the background; Multi-scale aggregation module; used to determine the location of the background based on the output of the mask-aware cross-attention module, significantly improving the model's expressive power by integrating multi-scale receptive fields; The mask-aware cross-attention module includes: The position embedding path module: Based on the input image layout block and the background mask, the position vectors of the input image layout block and the background mask are obtained through convolutional layers respectively; the input image layout block divides the input image into square blocks of the same size according to a preset resolution, and records the position of the square blocks after each division. The positions of the square blocks after division constitute the input image layout block. Content embedding path module: Based on the input image and background mask, it is used to obtain the input image content query vector, key vector and value vector respectively through convolutional layers, and the background mask is obtained by passing through convolutional layers to obtain the key vector and value vector of the background mask content; Combination module: Combines the input image layout block position vector and the background mask position vector of the output of the position embedding path with the input image content query vector, key vector and value vector of the output of the content embedding path to obtain the combined query vector, key vector and value vector; Cross-attention module: Based on the combined query vector, key vector and value vector output by the combined module, it combines information from different modalities to connect the target's location information and content information together; The multi-scale aggregation module includes: First branch: Used for input feature extraction; The second branch is used for extracting lateral features. The third branch is used for extracting vertical features; The fourth branch is used to preserve input features. Integration module: It is used to concatenate the features output from the first branch, the second branch and the third branch, then add the concatenated feature to the output of the fourth branch, and then output the feature.

[0009] Furthermore, the encoding / decoding structure with a semantic injection module includes an encoding / decoding structure generation body and a semantic injection module; The encoding / decoding generation main body includes: a four-layer downsampling module, an intermediate layer module, and a four-layer upsampling module; Four-layer downsampling module: used to map the image to the latent space; Intermediate layer module: used for feature extraction based on the mapped image output by the four-layer downsampling module; Semantic injection module: Based on the features extracted from the output of the intermediate module, it extracts the deep semantic information retained by the network, restricts the generation position of the target, and learns the positional relationship between the targets; The fourth-layer upsampling module is used to map the semantic modulation parameters output by the semantic injection module and the extracted features output by the intermediate layer module back to the graphics space.

[0010] Furthermore, the ship background fusion device includes: Fusion preprocessing module: used to perform a preliminary fusion operation on the ship target image output by the encoder-decoder structure and the background image output by the diffusion model to obtain preliminary fusion features; the fusion preprocessing module includes a single convolutional layer; The spectral attention module is used to obtain the query vector, key vector, and value vector of the preliminary fusion features based on the output of the fusion preprocessing module through a convolutional layer. Then, the query vector and key vector of the preliminary fusion features are mapped to the spectral space using FFT and multiplied. Finally, IFFT is used to map them to the temporal space and multiply them with the value vector of the preliminary fusion features to obtain further fusion features. This preserves global context information while providing a representation of frequency domain features. Fusion post-processing module: used to map the fusion features of the output of the spectral attention module to the graphics space to obtain the final fused image. The fusion post-processing module includes a single convolutional layer.

[0011] An apparatus for generating SAR images of ships against complex backgrounds, comprising: Acquisition module: Used to acquire tags that include at least ships and oceans; Modules: Used to build SAR ship image models with complex backgrounds and to generate SAR ship images with complex backgrounds; Training module: Used to train a SAR ship image model with complex backgrounds, and obtain a trained SAR ship image model with complex backgrounds; Implementation module: This module is used to input labels, including at least those for ships and oceans, into a trained complex background SAR ship image model, thereby generating complex background SAR ship images based on the input labels.

[0012] This invention provides a method for generating SAR ship images with complex backgrounds. It effectively addresses the problems of current SAR ship generation methods, such as lack of detailed modeling of complex scene elements, inability to reconstruct high-frequency features like coastal textures and terrain changes, and severe target omissions and misalignments in dense maritime scenes. Furthermore, by employing a label-before-image generation approach, it effectively saves researchers' time and provides technical support for SAR image processing algorithms. The method offers the following advantages: 1. To address the lack of detailed modeling for complex scene elements, this invention breaks down SAR image generation into three independent paths: ship generation, background generation, and ship-background fusion. Different models are selected for targets with different characteristics, and the output is finally integrated through ship-background fusion, effectively solving the problem of detailed modeling for complex scenes.

[0013] 2. To reconstruct high-frequency features such as coastal texture and topographic changes, this invention introduces a diffusion model with Mask-Aware Cross-Attention (MaCA) and Multi-Scale Aggregation (MSA) modules to generate the background. By fusing background content features and location features, and extracting and integrating information from different directions of the target, high-quality generation of the background target is achieved.

[0014] 3. To address the problem of severe target omission and misalignment in dense maritime scenarios, this invention introduces an encoder-decoder structure with a semantic injection module (SIM) to generate ship targets. By modeling the target mask and extracting deep information, accurate generation of ship targets is achieved. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart comparing the method of this application with the traditional method; Figure 2 This is a structural diagram of the method for generating SAR ship images with complex backgrounds; Figure 3 This is a structural diagram of the semantic injection module; Figure 4 This is a diagram of a mask-aware cross-attention structure; Figure 5 This is a multi-scale aggregation module structure diagram; Figure 6 This is a diagram of the spectrum attention module structure; Figure 7 The quality of the images generated by the model structure proposed in this invention is demonstrated by visualizing the SAR image generation results. Among them, (a) is a real image visualization label, (b) is a real image, (c) is an image generated by this invention, (d) is a LocSAR generated image, (e) is a SARGAN generated image, (f) is a LayoutDiffusion generated image, and (g) is a PIX2PIX generated image.

[0017] Figure 8 The following comparisons of detection performance based on the Oriented R-CNN model are presented: (a) Detection results after training with real images; (b) Detection results after training with images generated by PIX2PIX; (c) Detection results after training with images generated by LayoutDiffusion; (d) Detection results after training with images generated by LocSAR; (e) Detection results after training with images generated by SARGAN; (f) Detection results after training with images generated by CondFusionSAR. Figure 9 The results present a horizontal comparison of the detection performance based on the Oriented RepPoints model: (a) the detection results after training with real images; (b) the detection results after training with images generated by PIX2PIX; (c) the detection results after training with images generated by LayoutDiffusion; (d) the detection results after training with images generated by LocSAR; (e) the detection results after training with images generated by SARGAN; and (f) the detection results after training with images generated by CondFusionSAR. Detailed Implementation

[0018] It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0021] Figure 1 This is a flowchart comparing the method of this application with the traditional method; A method for generating complex background SAR ship images is disclosed. Based on input labels, a ship mask and a background mask are generated. Pixel-by-pixel operations are used to separate the ship target from the background. Then, a complex background SAR ship image generation model is constructed, comprising a diffusion model, an encoder-decoder structure, and a ship-background fusion processor. The diffusion model generates diverse background images through a forward diffusion network and a backward denoising network. The backward denoising network integrates a mask-aware cross-attention module and a multi-scale aggregation module to constrain the determined location of the background using mask information. The encoder-decoder structure includes a semantic injection module to learn the positional relationships between targets and generate ship target images. Finally, the generated ship target and background images are fused using a spectral attention module in the ship-background fusion processor to preserve global contextual information and enhance frequency domain feature representation, ultimately generating a high-quality complex background SAR ship image.

[0022] A method for generating SAR images of ships with complex backgrounds includes the following steps: S1: Obtain tags that include at least ships and the ocean; S2: Construct a SAR ship image model with complex backgrounds to generate SAR ship images with complex backgrounds; S3: Based on manually labeled tags and real images, train a SAR ship image model with complex background to obtain a well-trained SAR ship image model with complex background. S4: Input labels, including at least those for ships and oceans, into a trained complex background SAR ship image model to generate complex background SAR ship images based on the input labels.

[0023] Steps S1 / S2 / S3 / S4 are executed sequentially; In addition to ship tags and marine tags, the tags may also include land tags, lighthouse tags, etc. The complex background SAR ship image model includes: Decoupling module: Used to decouple based on the input labels to obtain an image containing only the ship target, an image containing only the background, and a mask image of the ship target; Diffusion model: Based on the background-only image output by the decoupling module, a background image is generated; Encoder-decoder structure: Based on the image containing only the ship target and the mask image of the ship target output by the decoupling module, the ship target image is generated; Ship background fusion unit: Based on the background image output by the diffusion model and the ship target image output by the encoder-decoder structure, it realizes the fusion of SAR images.

[0024] Furthermore, the label uses a rotated bounding box label, and the format of the rotated bounding box label is defined using OpenCV's definition method: including the x-coordinate of the center point, the y-coordinate of the center point, the length of the first side rotated counterclockwise around the horizontal axis, the length of the adjacent sides of the first side, and the angle between the horizontal axis and the first side rotated counterclockwise, the angle range of which is specified. ; The process of decoupling based on the input labels to obtain an image containing only the ship target, an image containing only the background, and a mask image of the ship target is as follows: The decoupling of ships and background targets in existing SAR images is performed using rotated bounding box labels. The specific process is as follows: The ship mask image is generated based on the rotated bounding box label, and the background mask image is obtained by inverting the ship mask image. The image containing only ships is obtained by multiplying the ship mask image and the original input image pixel by pixel. This operation retains the pixel values ​​of the ships while setting the pixel values ​​of the background to zero, thereby achieving effective separation between the ships and the background. The image containing only the background is obtained by multiplying the background mask and the original image pixel by pixel. For the resulting blank areas, the surrounding background pixel information is used to fill and repair them.

[0025] Furthermore, the diffusion model includes: diffusion-generated structural entities; The diffusion-generated structure includes: Forward diffusion network: used to gradually add random noise to an image that contains only background based on the output of the decoupling module, eventually obtaining a noisy image; Backward denoising network: Based on the full-noise image output by the forward diffusion network, it predicts the noise added at each step through training and removes it from the full-noise image to restore the input image.

[0026] Furthermore, the backward denoising network includes a mask-aware cross-attention module and a multi-scale aggregation module; The mask-aware cross-attention module (MaCA) includes: The position embedding path module: Based on the input image layout block and the background mask, the position vectors of the input image layout block and the background mask are obtained through convolutional layers respectively; the input image layout block divides the input image into square blocks of the same size according to a preset resolution, and records the position of the square blocks after each division. The positions of the square blocks after division constitute the input image layout block. The content embedding path module is based on the input image and the background mask. After passing through convolutional layers, it obtains the input image content query vector, key vector, and value vector, respectively. The background mask is obtained by passing through convolutional layers to obtain the key vector and value vector of the background mask content. It consists of location embedding path, content embedding path, combined module and cross attention module; Combination module: Combines the input image layout block position vector and the background mask position vector of the output of the position embedding path with the input image content query vector, key vector and value vector of the output of the content embedding path to obtain the combined query vector, key vector and value vector; Cross-attention module: Based on the combined query vector, key vector and value vector output by the combined module, it combines information from different modalities to connect the target's location information and content information together; The Multiscale Aggregation Module (MSA) includes: First branch: Used for input feature extraction; The second branch is used for extracting lateral features. The third branch is used for extracting vertical features; The fourth branch is used to preserve input features. Integration module: It is used to concatenate the features output from the first branch, the second branch and the third branch, then add the concatenated feature to the output of the fourth branch, and then output the feature.

[0027] Furthermore, the encoding / decoding structure with a semantic injection module includes an encoding / decoding structure generation body and a semantic injection module; The encoding / decoding generation main body includes: a four-layer downsampling module, an intermediate layer module, and a four-layer upsampling module; Four-layer downsampling module: used to map the image to the latent space; Intermediate layer module: used for feature extraction based on the mapped image output by the four-layer downsampling module; Semantic Injection Module (SIM): Based on the extracted features output by the intermediate module, it extracts the deep semantic information retained by the network, restricts the generation position of the target, and learns the positional relationship between the targets. The fourth-layer upsampling module is used to map the semantic modulation parameters output by the semantic injection module and the extracted features output by the intermediate layer module back to the graphics space.

[0028] Furthermore, the ship background fusion device includes: Fusion preprocessing module: used to perform a preliminary fusion operation on the ship target image output by the encoder-decoder structure and the background image output by the diffusion model to obtain preliminary fusion features; the fusion preprocessing module includes a single convolutional layer; The Spectrum Attention Module (SAM) is used to obtain the query vector, key vector, and value vector of the preliminary fusion features based on the output of the fusion preprocessing module through a convolutional layer. Then, the query vector and key vector of the preliminary fusion features are mapped to the spectral space using FFT and multiplied. Finally, IFFT is used to map them to the time domain space and multiply them with the value vector of the preliminary fusion features to obtain further fusion features. This preserves global context information while providing a representation of frequency domain features. Fusion post-processing module: used to map the fusion features of the output of the spectral attention module to the graphics space to obtain the final fused image. The fusion post-processing module includes a single convolutional layer.

[0029] An apparatus for generating SAR images of ships against complex backgrounds, comprising: Acquisition module: Used to acquire tags that include at least ships and oceans; Modules: Used to build SAR ship image models with complex backgrounds and to generate SAR ship images with complex backgrounds; Training module: Used to train a SAR ship image model with complex backgrounds, and obtain a trained SAR ship image model with complex backgrounds; Implementation module: This module is used to input labels, including at least those for ships and oceans, into a trained complex background SAR ship image model, thereby generating complex background SAR ship images based on the input labels.

[0030] Example 1: A method for generating SAR images of ships with complex backgrounds includes the following steps: S1: Obtain tags that include at least ships and the ocean; S2: Construct a SAR ship image model with complex backgrounds to generate SAR ship images with complex backgrounds; S3: Train the SAR ship image model with complex background to obtain a trained SAR ship image model with complex background. S4: Input labels, including at least those for ships and oceans, into a trained complex background SAR ship image model to generate complex background SAR ship images based on the input labels.

[0031] The complex background SAR ship image model includes: Decoupling module: Used to decouple based on the input labels to obtain an image containing only the ship target, an image containing only the background, and a mask image of the ship target; Diffusion model: Based on the background-only image output by the decoupling module, a background image is generated; Encoder-decoder structure: Based on the image containing only the ship target and the mask image of the ship target output by the decoupling module, the ship target image is generated; Ship background fusion unit: Based on the background image output by the diffusion model and the ship target image output by the encoder-decoder structure, it realizes the fusion of SAR images.

[0032] The process of decoupling based on the input labels to obtain an image containing only the ship target, an image containing only the background, and a mask image of the ship target is as follows: like Figure 2 As shown, the input consists of manually labeled tags and real images. A mask is generated for each category of target using coordinate analysis. Specifically, the target label area is set to 1, and the rest to 0, forming a black and white ship mask. .

[0033] The results of coordinate analysis are determined by formulas (1), (2), (3) and (4).

[0034] (1) (2) (3) (4) in, These represent the x-coordinate of the center point of the rotating frame, the y-coordinate of the center point, the length of the first side rotated counterclockwise by the horizontal axis, the length of the other side, and the angle between the horizontal axis and the first side rotated counterclockwise.

[0035] After obtaining the ship mask, a pixel-by-pixel multiplication operation is performed with the original input image, retaining the ship's pixel values ​​while setting the background pixel values ​​to zero, thereby achieving effective separation between the ship and the background. This process can be expressed as formula (5): (5) in, This represents element-wise multiplication. This represents the input image.

[0036] The background mask is obtained by inverting the ship mask, and this process can be expressed as formula (6): (6) An image containing only the background is obtained by element-wise multiplication of the original image and the background mask. Blank areas resulting from the product are filled using information from the surrounding background pixels.

[0037] (7) in, This represents element-wise multiplication.

[0038] The layout blocks of the input image are generated based on three preset resolutions (16, 32, 64). For an input image of size (3, 512, 512), we divide it into blocks of 16×16, 32×32, and 64×64 respectively. Specifically, at a resolution of 16, the image is divided into a 16×16 grid, where the position information of each block (e.g., the spatial coordinates of the block in the nth row and mth column) is specified. This will be recorded and used for the generation of subsequent background targets.

[0039] The diffusion model includes: a diffusion-generated structural entity; The diffusion-generated structure includes: Forward diffusion network: used to gradually add random noise to an image that only contains the background based on the output of the decoupling module, eventually obtaining a noisy image; it belongs to the forward diffusion process; Backward denoising network: Based on the full-noise image output by the forward diffusion network, it predicts the noise added at each step through training and removes it from the full-noise image to restore the input image. It belongs to the reverse denoising process.

[0040] like Figure 2 As shown in the background generation branch, during the forward diffusion process, the image containing only the background is first mapped to the latent space by a pre-trained encoder. Then, noise is gradually added to the input image at each time step, ultimately resulting in a fully noisy image. In the reverse denoising process, a noise prediction network predicts the noise to be removed at each time step, thus transforming the entire noisy image... The image is then restored to its original state as the input image. The noise prediction network is then executed by an improved denoising UNet network.

[0041] The UNet network generally follows a U-shaped structure of encoder (downsampling) - decoder (upsampling), but adds mask-aware cross attention (MaCA) and multi-scale aggregation (MSA) modules on the critical path of its data flow.

[0042] The mask-aware cross-attention module is integrated into the input and output layers of the UNet network. For example... Figure 3 As shown, this module takes the previous background noise map, the mask map of the background target, and the input image layout blocks as input, and consists of three parts: the position embedding path, the content embedding path, and the combination module. The input of the position embedding path is a set of mask maps of the background target of the image layout blocks at different resolutions. After passing through convolutional layers, the input image layout block position vectors of sizes (16, 32, 4096) are obtained respectively. The position vector of the background mask of size (16, 32, 10) The input to the content embedding path consists of the background noise image and the mask image of the background target from the previous time step. After passing through convolutional layers, these elements yield input image content query vectors of size (16, 32, 4096). Key vector Value vector Key vectors of the content of a background mask of size (16, 32, 10) and The value vector. The combination module takes the outputs of the content embedding path and the location embedding path as input to derive the combined query vector. Value vector With key vector Specifically, the input image content query vector With input image layout block position vector Combining along the channel direction yields a combined query vector of size (16, 64, 4096). Input image content key vector With background mask content value vector Combining along the channel direction yields an intermediate result of size (16, 64, 4096), which is then compared with the key vector of the background mask content. and position vector The concatenated result (16, 64, 10) along the channel dimension is concatenated along the height, resulting in a combined key vector of size (16, 64, 4106). The value vector of the input image content. Value vector of background mask content By concatenating along the height, we obtain a combined value vector of size (16, 32, 4106). Finally, the cross-attention mechanism is used to... , and Integration is achieved through formula (8).

[0043] (8) Multi-scale aggregation (MSA) modules are integrated into the middle layer of the UNet network; such as Figure 4 As shown, this module takes the output of the downsampling layer of the UNet network as input and is divided into four different branches to process the input features. Branch 1 consists of a 1×1 standard convolution with a stride of 1 and a 3×3 standard convolution with a stride of 1, transforming the input of size (2, 1024, 16, 16) into the output of branch 1 of size (2, 256, 16, 16). Branch 2 consists of a 1×1 standard convolution with a stride of 1, a 3×3 dilated convolution with a stride of 1 and an inflation rate of 5, a 1×3 standard convolution with a stride of 1, and a 3×1 standard convolution with a stride of 1. It extracts horizontal features and transforms the input of size (2, 1024, 16, 16) into the output of branch 2, which is (2, 256, 16, 16). Branch 2 consists of a 1×1 standard convolution with a stride of 1, a 3×3 dilated convolution with a stride of 1 and an inflation rate of 5, a 3×1 standard convolution with a stride of 1, and a 1×3 standard convolution with a stride of 1. It extracts vertical features and transforms the input of size (2, 1024, 16, 16) into the (2, 256, 16, 16) output of Branch 2. The four-branch convolution consists of only one 1×1 standard convolution with a stride of 1, preserving the features of the input. The final output of this module is determined by formula (9): (9) in, This represents the stacking operation in vectors.

[0044] like Figure 2 As shown in the ship generation branch, the ship mask is adjusted to a size of 512×512, and then channel adjustment is performed through a 7×7 convolution to generate a lower-dimensional feature map. This feature map input is downsampled through four 3×3 convolutions with a stride of 2 to obtain a high-dimensional feature map of 256×32×32. Nine residual blocks are used to extract features in the high dimension, and then upsampled through four 3×3 convolutions with a stride of 2 arranged in the reverse order of the downsampling process to complete the reconstruction of the low-dimensional feature map of the image.

[0045] The semantic injection module is added during the upsampling process, and its structure is as follows: Figure 5 As shown.

[0046] When using residual blocks for high-dimensional feature extraction, the extracted features are retained at each step. This semantic information is input into the semantic injection module, decoded through three 3×3 convolutions with a stride of 1, and then scaled and offset based on the decoded information to obtain a low-dimensional feature map containing semantic relationships. Finally, the output network performs channel adjustment to obtain a 512×512 generated ship image. During training, the generated ship image is also input into the discriminator network, undergoing four 4×4 convolutions to obtain the discrimination result, improving the quality of the generated image.

[0047] like Figure 2 As shown in the ship background fusion diagram, this part takes the outputs of the background generation branch and the ship generation branch as inputs. The outputs of each branch first undergo a fusion preprocessing process, which consists of a 3×3 convolutional layer that maps the outputs of each branch to the latent space, and then sums them to obtain preliminary fused features.

[0048] Spectrum attention module, such as Figure 6 As shown, the initial fusion features are first projected to generate corresponding query vectors, key vectors, and value vectors. Subsequently, the query vectors and key vectors are mapped to the frequency domain via Fast Fourier Transform (FFT) and multiplied in the frequency domain to capture global frequency domain correlations. The calculation results are then mapped back to the time domain via Inverse Fast Fourier Transform (IFFT), and multiplied with the value vector to ultimately generate deeper fusion features.

[0049] The post-fusion processing section also consists of a 3×3 convolutional layer with a stride of 1. The deep fusion features are mapped to the image space by the post-fusion processing section to obtain the final SAR generated map.

[0050] For the loss function in the network, this invention uses three parts: ship generation loss, etc. Background generation loss fusion loss . and The calculation is performed using formulas (10) and (11): (10) in, Represents the generated image and the input image. Represents the discriminator.

[0051] (11) in This represents the latent variables obtained from the original image by the encoder. Indicates the time step. This represents standard Gaussian noise. It was after The latent variables generated after the addition of step noise are produced by the forward process. Indicates conditional information. This represents the prediction noise during the reverse process.

[0052] Fusion loss It consists of two parts, namely L1 loss and perceptual loss, which are calculated by formulas (12) and (13): (12) (13) in Represents the generated image and the input image. This represents a pre-trained VGG network. Indicates the selection of the VGG network. Feature extraction is performed on the layer. and These represent the number of channels, height, and width of the feature, respectively.

[0053] The final fusion loss is obtained from equation (14): (14) in The weight representing the impact of perceived loss on total loss is set to 0.5 in our model.

[0054] To demonstrate that the method for generating SAR ship images with complex backgrounds described in this invention can generate high-quality SAR images, the generation results were compared with those of PIX2PIX, LayoutDiffusion, LocSAR, and SARGAN. The results are shown in Table 1. Table 1. Comparison of different generative models

[0055] Experimental results demonstrate that the model structure proposed in this invention can improve the quality of the generated images. For example... Figure 7 As shown, the quality of the images generated by the model structure proposed in this invention is demonstrated by visualizing the SAR image generation results. (a) is a real image visualization label, (b) is a real image, (c) is an image generated by this invention, (d) is a LocSAR generated image, (e) is a SARGAN generated image, (f) is a LayoutDiffusion generated image, and (g) is a PIX2PIX generated image.

[0056] To further verify the applicability and value of the images generated by the proposed generative structure in other SAR image processing tasks, we constructed a hybrid training set using 132 images generated by each generative model under the same conditions and 132 real SAR images, respectively, to train two mainstream detection models: Oriented R-CNN and Oriented reppoints. The comparison of the models' detection performance under identical training conditions is shown in the table below: Table 2. Detection results of SAR images generated based on different models

[0057] Figure 8 and Figure 9 The test results were visualized. Figure 8 The results present a horizontal comparison of the detection performance based on the Oriented R-CNN model: (a) is the detection result after training with real images; (b) is the detection result after training with images generated by PIX2PIX; (c) to (f) are the detection results after training with images generated by LayoutDiffusion, LocSAR, SARGAN and CondFusionSAR, respectively. Figure 9 The results present a horizontal comparison of the detection performance based on the Oriented RepPoints model: (a) is the detection result after training with real images; (b) is the detection result after training with images generated by PIX2PIX; (c) to (f) are the detection results after training with images generated by LayoutDiffusion, LocSAR, SARGAN and CondFusionSAR, respectively.

[0058] comprehensive Figure 8 , Figure 9 Analysis of the visualization results and the quantitative data in Table 2 shows that the images synthesized using the generated structure proposed in this invention can substantially improve the generalization ability of the detection model.

[0059] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating SAR images of ships against complex backgrounds, characterized in that, Includes the following steps: Obtain tags that include at least ships and the ocean; Construct a SAR ship image model with complex backgrounds to generate SAR ship images with complex backgrounds; Based on manually labeled tags and real images, a SAR ship image model with complex backgrounds is trained to obtain a well-trained SAR ship image model with complex backgrounds. By inputting labels that include at least ships and ocean into a trained SAR ship image model with complex backgrounds, SAR ship images with complex backgrounds can be generated based on the input labels.

2. The method for generating SAR ship images with complex backgrounds according to claim 1, characterized in that, The complex background SAR ship image model includes: Decoupling module: Used to decouple based on the input labels to obtain an image containing only the ship target, an image containing only the background, and a mask image of the ship target; Diffusion model: Based on the background-only image output by the decoupling module, a background image is generated; Encoder-decoder structure: Based on the image containing only the ship target and the mask image of the ship target output by the decoupling module, the ship target image is generated; Ship background fusion unit: Based on the background image output by the diffusion model and the ship target image output by the encoder-decoder structure, it realizes the fusion of SAR images.

3. The method for generating SAR ship images with complex backgrounds according to claim 1, characterized in that, The label uses a rotated bounding box label, and its format is defined using OpenCV: It includes the x-coordinate of the center point, the y-coordinate of the center point, the length of the first edge rotated counter-clockwise around the horizontal axis, the lengths of the adjacent edges of the first edge, and the angle between the horizontal axis and the first edge rotated counter-clockwise. The angle range is specified in the OpenCV definition. ; The process of decoupling based on the input labels to obtain an image containing only the ship target, an image containing only the background, and a mask image of the ship target is as follows: The decoupling of ships and background targets in existing SAR images is performed using rotated bounding box labels. The specific process is as follows: The ship mask image is generated based on the rotated bounding box label, and the background mask image is obtained by inverting the ship mask image. The image containing only ships is obtained by multiplying the ship mask image and the original input image pixel by pixel. This operation retains the pixel values ​​of the ships while setting the pixel values ​​of the background to zero, thereby achieving effective separation between the ships and the background. The image containing only the background is obtained by multiplying the background mask and the original image pixel by pixel. For the resulting blank areas, the surrounding background pixel information is used to fill and repair them.

4. The method for generating SAR ship images with complex backgrounds according to claim 2, characterized in that, The diffusion model includes: a diffusion-generated structural entity; The diffusion-generated structure includes: Forward diffusion network: used to gradually add random noise to an image that contains only background based on the output of the decoupling module, eventually obtaining a noisy image; Backward denoising network: Based on the full-noise image output by the forward diffusion network, it predicts the noise added at each step through training and removes it from the full-noise image to restore the input image.

5. The method for generating SAR ship images with complex backgrounds according to claim 4, characterized in that, The backward denoising network includes: Mask-aware cross-attention module: used to acquire mask-related information and generate the determined position of the background; Multi-scale aggregation module; used to determine the location of the background based on the output of the mask-aware cross-attention module, significantly improving the model's expressive power by integrating multi-scale receptive fields; The mask-aware cross-attention module includes: The position embedding path module: Based on the input image layout block and the background mask, the position vectors of the input image layout block and the background mask are obtained through convolutional layers respectively; the input image layout block divides the input image into square blocks of the same size according to a preset resolution, and records the position of the square blocks after each division. The positions of the square blocks after division constitute the input image layout block. Content embedding path module: Based on the input image and background mask, it is used to obtain the input image content query vector, key vector and value vector by splitting. The background mask is split into key vector and value vector of the background mask content after passing through a convolutional layer. Combination module: Combines the input image layout block position vector and the background mask position vector of the output of the position embedding path with the input image content query vector, key vector and value vector of the output of the content embedding path to obtain the combined query vector, key vector and value vector; Cross-attention module: Based on the combined query vector, key vector and value vector output by the combined module, it combines information from different modalities to connect the target's location information and content information together; The multi-scale aggregation module includes: First branch: Used for input feature extraction; The second branch is used for extracting lateral features. The third branch is used for extracting vertical features; The fourth branch is used to preserve input features. Integration module: It is used to concatenate the features output from the first branch, the second branch and the third branch, then add the concatenated feature to the output of the fourth branch, and then output the feature.

6. The method for generating SAR ship images with complex backgrounds according to claim 1, characterized in that, The encoding / decoding structure with a semantic injection module includes an encoding / decoding structure generation body and a semantic injection module; The encoding / decoding generation main body includes: a four-layer downsampling module, an intermediate layer module, and a four-layer upsampling module; Four-layer downsampling module: used to map the image to the latent space; Intermediate layer module: used for feature extraction based on the mapped image output by the four-layer downsampling module; Semantic injection module: Based on the features extracted from the output of the intermediate module, it extracts the deep semantic information retained by the network, restricts the generation position of the target, and learns the positional relationship between the targets; The fourth-layer upsampling module is used to map the semantic modulation parameters output by the semantic injection module and the extracted features output by the intermediate layer module back to the graphics space.

7. The method for generating SAR ship images with complex backgrounds according to claim 1, characterized in that, The ship background fusion device includes: Fusion preprocessing module: used to perform a preliminary fusion operation on the ship target image output by the encoder-decoder structure and the background image output by the diffusion model to obtain preliminary fusion features; the fusion preprocessing module includes a single convolutional layer; The spectral attention module is used to obtain the query vector, key vector, and value vector of the preliminary fusion features based on the output of the fusion preprocessing module through a convolutional layer. Then, the query vector and key vector of the preliminary fusion features are mapped to the spectral space using FFT and multiplied. Finally, IFFT is used to map them to the temporal space and multiply them with the value vector of the preliminary fusion features to obtain further fusion features. This preserves global context information while providing a representation of frequency domain features. Fusion post-processing module: used to map the fusion features of the output of the spectral attention module to the graphics space to obtain the final fused image. The fusion post-processing module includes a single convolutional layer.

8. A device for generating SAR images of ships against complex backgrounds, characterized in that, include: Acquisition module: Used to acquire tags that include at least ships and oceans; Modules: Used to build SAR ship image models with complex backgrounds and to generate SAR ship images with complex backgrounds; Training module: Used to train a SAR ship image model with complex backgrounds, and obtain a trained SAR ship image model with complex backgrounds; Implementation module: This module is used to input labels, including at least those for ships and oceans, into a trained complex background SAR ship image model, thereby generating complex background SAR ship images based on the input labels.

Images