Remote sensing image style migration method based on DualDepthResGAN model
By introducing a resolution adjustment module and residual connection into the DualGAN model and combining it with depth information, the scale inconsistency and gradient vanishing problems in remote sensing image style transfer are solved, and the retention of structural information and the learning of target style are achieved during the remote sensing image style transfer process.
Patent Information
- Application Number
- CN202410430645.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-10
- Publication Date
- 2025-10-17
AI Technical Summary
The existing DualGAN model suffers from scale inconsistency and gradient vanishing problems in remote sensing image style transfer, and is unable to effectively preserve the structural information of real images.
A resolution adjustment module and residual connection are added to the generator part of the DualGAN model, and depth information is integrated to construct the DualDepthResGAN model. The training process is optimized through multiple loss functions such as deep supervision loss and cycle consistency loss to achieve remote sensing image style transfer.
In the process of remote sensing image style transfer, it can effectively learn the target style and retain the original structural information, solve the problems of scale inconsistency and gradient disappearance, and improve the quality of image style transfer.
Smart Images
Figure CN120807268A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application relates to a remote sensing image style transfer method based on a DualDepthResGAN model and belongs to the field of image processing. BACKGROUND
[0002] DualGAN is a generative adversarial network (GAN) based image style transfer method, which aims to realize the conversion of image style. It is composed of two generator networks and two discriminator networks, each generator network is responsible for converting the input image from one domain to another domain, and each discriminator network is responsible for distinguishing whether the generated image is from the target domain. By alternately training the generator and discriminator networks, DualGAN can learn an effective mapping relationship between the two domains, thereby realizing the conversion of image style. This method shows the advantages of adversarial training in experiments, and can generate images with high-fidelity style while retaining the content information of the original image.
[0003] However, compared with ordinary images, remote sensing images contain more complex geographical and spectral information. Directly applying DualGAN to remote sensing image style transfer will have the following problems: first, the distance from any object in a single domain of remote sensing images to the camera remains constant, resulting in some scale-invariant classes in remote sensing images having relatively fixed sizes. When two remote sensing images have different resolutions, the scale-invariant classes may be different. This will cause the problem of scale incoordination in the generated image when remote sensing images are subjected to style transfer, and different resolutions input into the same discriminator for discrimination are easy to cause the phenomenon of gradient disappearance. Second, remote sensing image style transfer belongs to the category of real image to real image conversion. In the process of image to image conversion, the network should modify the real image less than the synthetic image, and the structure information of the real image will not be modified in the conversion process.
[0004] To solve this problem, the generator structure is improved on the DualGAN network model and the depth information is fused, and a new remote sensing image style transfer method, i.e. a remote sensing image style transfer method based on a DualDepthResGAN model, is proposed. This method can retain the original structure information while fully learning the target style when performing remote sensing image style transfer. SUMMARY
[0005] The present application aims at the limitations of the DualGAN model in remote sensing image style transfer, and proposes a remote sensing image style transfer method based on the DualDepthResGAN model. This method adds a resolution adjustment module and a residual connection to the generator part of the DualGAN model, and fuses depth information to preserve the original structural information while fully learning the target style in remote sensing image style transfer.
[0006] The technical solutions of the present application are as follows:
[0007] The remote sensing image style transfer method based on the DualDepthResGAN model is a style transfer method for remote sensing images, and the implementation steps are as follows:
[0008] (1) Prepare the source domain data set X S and the target domain data set X T and their corresponding DSM image information set Z S and Z T , and initialize.
[0009] (2) Send X s into the generator ResoG S→T to generate X S→T and Z′ S , and send X T into the generator ResoG T→S to generate X T→S and Z′ T .
[0010] (3) Compare Z′ S and Z S to generate depth supervision loss L DSL (S), and compare Z′ T and Z T to generate depth supervision loss L DSL (T).
[0011] (4) Send X S→T into the generator ResoG T→S to generate X′ S and Z S→T , and send X T→s into the generator ResoG S→T to generate X′ T and Z T→S .
[0012] (5) Compare X′ S and X S , X′ T and X T respectively to generate cycle consistency loss L cyc (S, T), Lcyc (T, S).
[0013] (6) Compare Z s→T and Z s , Z T→S and Z T Generate deep cycle consistent loss L DCCL (S), L DCCL (T).
[0014] (7) Update the parameters of ResoG DSL (S), ResoG cyc (S, T), ResoG DCCL (S) and ResoG DSL (T), ResoG cyc (T, S), ResoG DCCL (T) through L S→T and ResoG T→S .
[0015] (8) Send X S→T and X T to the discriminator D T to generate adversarial loss L adv (T), while sending X T→S and X s to the discriminator D S to generate adversarial loss L adv (S).
[0016] (9) Update the parameters of the discriminators D adv (S) and D adv (T) through L S and D T .
[0017] (10) Check whether the total loss function L total reaches equilibrium or the number of iterations reaches the maximum value, and jump to step (11) if any of the above conditions is met, and jump to step (2) if none of the above conditions is met.
[0018] (11) Perform image style transfer.
[0019] The beneficial effects of the present application are as follows:
[0020] The remote sensing image style transfer method based on DualDepthResGAN can fully retain the original structural information under the premise of considering the high resolution, rich feature of ground objects, fixed size and other characteristics of remote sensing images, and learn the image features of the target domain, thereby providing strong support for the current unsupervised field adaptive remote sensing image semantic segmentation field. BRIEF DESCRIPTION OF DRAWINGS
[0021] Figure 1Algorithm flow chart of remote sensing image style transfer based on DualDepthResGAN
[0022] Figure 2 Structure diagram of generator ResoG
[0023] Figure 3 Algorithm flow chart of generator ResoG generating target style image and DSM image DETAILED DESCRIPTION
[0024] The present application will be further described in detail below in conjunction with the accompanying drawings.
[0025] As Figure 1 Algorithm flow chart of remote sensing image style transfer based on DualDepthResGAN. The implementation steps of the algorithm of remote sensing image style transfer based on DualDepthResGAN are as follows:
[0026] 1. Prepare source domain data set X S and target domain data set X T and their corresponding DSM image information set Z S and Z T , and initialize.
[0027] 2. Send X S into generator ResoG S→T to generate X S→T and Z S→T , and send X T into generator ResoG T→S to generate X T→S and Z T→S . The structure of generator ResoG is shown in Figure 2 .
[0028] 3. Compare Z' S and Z S to generate depth supervision loss L DSL (S), and compare Z' T and Z T to generate depth supervision loss L DSL (T). The calculation formula is as follows:
[0029]
[0030]
[0031] 4. Send X S→T into generator ResoG T→S to generate X' S and Z S→T , and send X T→S into generator ResoGS→T Generate X' T and Z T→S .
[0032] 5. Compare X' S and X S , X' T and X T to generate cycle-consistent loss L cyc (S, T), L cyc (T, S). The calculation formula is as follows:
[0033]
[0034]
[0035] 6. Compare Z S→T and Z S , Z T→S and Z T to generate depth cycle-consistent loss L DCCL (S), L DCCL (T). The calculation formula is as follows:
[0036]
[0037]
[0038] 7. Update the parameters of ResoG DSL (S), ResoG cyc (S, T), ResoG DCCL (S) and ResoG DSL (T), ResoG cyc (T, S), ResoG DCCL (T) through L S→T and ResoG T→S .
[0039] 8. Send X S→T and X T to discriminator D T to generate adversarial loss L adv (T), and send X T→S and X S to discriminator D S to generate adversarial loss L adv (S). The calculation formula is as follows:
[0040]
[0041]
[0042] 9. Update the parameters of ResoG adv (S) and ResoG adv(T) update the discriminator D S and D T parameters.
[0043] 10. Check whether the total loss function or the number of iterations reaches a threshold value, if any of them is met, jump to step (11), if none of them is met, jump to step (2). The total loss function is calculated as follows:
[0044] L total = L adv (S, T) + L adv (T, S)
[0045] + 10 x (L cyc (S, T) + L cyc (T, S))
[0046] + 2 x (L DSL (S) + L DSL (T))
[0047] + L DCCL (S, T) + L DCCL (T, S)
[0048] As Figure 3 the algorithm flowchart for generating target style images and DSM images of ResoG, the implementation steps of the algorithm for generating images are as follows:
[0049] 1. Input image x, get feature representation X through encoder E.
[0050] 2. Input the feature representation X into the decoder D tr , D DSM , respectively, to get the residual term G(x), the DSM generated image z.
[0051] 3. Perform residual connection on the image x and the residual term G(x) to get the generated image temporary variable x'. The calculation formula is as follows:
[0052] x' = x + G(x)
[0053] 4. Input the generated image temporary variable x' into the resolution adjustment module to get the final target style image y. The expression is as follows:
[0054] y = resize(x')
[0055] Wherein the resize() function is used for resolution adjustment, which is realized by bicubic interpolation.
[0056] 5. Output the DSM generated image z and the target style image y.
Claims
1. A remote sensing image style transfer method based on the DualDepthResGAN model, characterized in that: When performing style transfer for remote sensing images, the generator module uses a residual connection to add the original image information to the residual term generated by the generator, and uses a bicubic interpolation algorithm to adaptively adjust the resolution to obtain the final generated image. The depth information provided by the DSM is introduced as a constraint. When generating a generated image with the target style, a corresponding DSM information image is additionally generated and compared with the original DSM image to optimize the generator parameters. Finally, when the generator and discriminator reach optimality or the training round limit is reached, the remote sensing image style transfer is executed.
2. According to claim 1, a remote sensing image style transfer method based on the DualDepthResGAN model is characterized in that: The generator of the generative adversarial network does not directly generate the target image but generates a residual term and performs a residual connection with the source image to obtain the target image.
3. According to claim 1, a remote sensing image style transfer method based on the DualDepthResGAN model is characterized in that: Adaptively adjust the resolution of remote sensing images to the target domain using bicubic interpolation.
4. According to claim 1, a remote sensing image style transfer method based on the DualDepthResGAN model is characterized in that: The depth information provided by DSM is introduced, and the relevant parameters of the generator are further optimized by comparing the DSM image generated by the generator with the original DSM image information.