Intelligent Generation Method of Satellite ISAR Image Data Based on Diffusion Model
Through the intelligent generation method based on the diffusion model, the latent spatial features of optical-ISAR image data are extracted and reverse diffusion and denoising are performed, which solves the problem of difficulty in obtaining ISAR image data in satellite targets, and realizes high-precision ISAR image generation and improves the image Generation quality and detection capability.
Patent Information
- Application Number
- CN202411170890.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-08-26
AI Technical Summary
Due to factors such as strong environmental confrontation, high target dynamics, and low accuracy of calculation instruments, satellite target ISAR image data is difficult to obtain, and only relatively complete optical image data can be detected, and high-precision ISAR image data is lacking.
Using the intelligent generation method of satellite ISAR image data based on diffusion model, the latent spatial feature extraction and generation of optical-ISAR image data through variational autoencoder and VQGAN model, and the Brown Bridge diffusion model is used to realize the use of the Brown Bridge diffusion model from the optical image domain to the ISAR image domain The latent space feature fusion and reverse diffusion denoising are generated to generate high-precision ISAR images.
High-precision conversion between optical images and ISAR images is realized, which avoids training instability and mode crash problems, improves the image generation quality of satellite targets, and enhances the detection ability of target information.
Smart Images

Figure CN118691463B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of radar image processing and computer vision technology, and in particular to a method for intelligently generating satellite ISAR image data based on a diffusion model. Background Art
[0002] The rapid development of artificial intelligence has a profound impact on the transformation of cutting-edge aerospace technologies. Data-driven deep learning technology has been applied effectively in tasks such as satellite target recognition, target reconstruction, and key component detection. However, such methods have high requirements for the quantity and quality of satellite target data in practical applications. ISAR images are one of the most widely used satellite target remote sensing sources. They can break through the limitations of optical images, improve the all-weather monitoring capabilities, elevation measurement capabilities, and multi-angle observation capabilities in the aerospace field, and enhance the detection of target information and aviation environment. However, due to factors such as strong environmental confrontation, high target dynamics, and low precision of measurement instruments, ISAR image data of satellite targets is difficult to obtain, and only relatively complete optical image data can be detected. Summary of the invention
[0003] Based on this, it is necessary to provide a satellite ISAR image data intelligent generation method based on a diffusion model that can realize the conversion between optical images and ISAR images in order to address the above technical problems.
[0004] A method for intelligently generating satellite ISAR image data based on a diffusion model, the method comprising:
[0005] Acquire optical-ISAR image data; construct an intelligent generation model for satellite ISAR image data; the intelligent generation model for satellite ISAR image data includes a latent space feature extraction module and an ISAR image generation module based on a diffusion model;
[0006] In the latent space feature extraction module, the optical-ISAR image data is encoded into the latent space according to the variational autoencoder to obtain the feature vector; two VQGAN models are pre-trained in the optical image domain and the ISAR image domain respectively, and the latent space features of the feature vector are extracted according to the VQGAN model to obtain the latent space features of the optical image domain and the ISAR image domain;
[0007] In the ISAR image generation module based on the diffusion model, the data distribution function and iterative process of the forward diffusion process of the latent space features from the ISAR image domain to the optical image domain are designed according to the Brownian bridge diffusion model to complete the latent space feature fusion, and the fused latent space features are subjected to reverse diffusion denoising to obtain the reverse diffusion features;
[0008] The inverse diffusion features are input into the latent space feature extraction module for decoding to obtain the reconstructed ISAR image;
[0009] The satellite ISAR image data intelligent generation model is trained by using optical-ISAR image data, a pre-set loss function for latent space feature extraction in the optical image domain, and a loss function for latent space feature extraction in the ISAR image domain, and optimizing the lower bound of evidence as the objective function of the reverse diffusion process, and a trained satellite ISAR image data intelligent generation model is obtained.
[0010] According to the trained satellite ISAR image data intelligent generation model, the input optical domain satellite image data is used to generate images to obtain satellite ISAR images.
[0011] In one embodiment, latent space features are extracted from feature vectors according to the VQGAN model to obtain latent space features in the optical image domain and the ISAR image domain, including:
[0012] According to the VQGAN model, the latent space features of the feature vector are extracted, and the latent space features of the optical image domain and the ISAR image domain are obtained as follows:
[0013] ;
[0014] in, , represents the input ISAR image and optical image, , represents the latent space features extracted from ISAR images and the latent space features extracted from optical images, , Represents the parameters of the VQGAN model in the ISAR image domain and the optical image domain.
[0015] In one embodiment, the loss function for latent space feature extraction in the optical image domain is:
[0016] ;
[0017] in, represents the reconstructed optical image, represents the encoder in the optical image domain, represents the decoder in the optical image domain, represents the fine-tunable weighting parameters in the optical image domain, represents the "stop gradient" operation, Representing latent space features of optical image data.
[0018] In one embodiment, the loss function for latent space feature extraction in the ISAR image domain is:
[0019] ;
[0020] in, represents the reconstructed ISAR image, represents the encoder in the ISAR image domain, represents the decoder in the ISAR image domain, represents the fine-tunable weighting parameters in the ISAR image domain, represents the "stop gradient" operation, Represents the latent space characteristics of ISAR image data.
[0021] In one embodiment, the data distribution function of the forward diffusion process is:
[0022] ;
[0023] in, represents the latent space features of optical image data, represents the latent space characteristics of ISAR image data, represents the diffusion time step The latent space features of the corresponding image, represents the diffusion time step, Indicates that the distribution function satisfies the normal distribution relationship. represents the total diffusion step length, Represents the identity matrix.
[0024] In one embodiment, the iterative process of the forward diffusion process is:
[0025] ;
[0026] in, , , , Represents variance.
[0027] In one embodiment, the objective function of the reverse diffusion process is
[0028] ;
[0029] in, represents the latent space features of optical image data, represents the latent space characteristics of ISAR image data, represents the latent space features of the image corresponding to the diffusion time step t, represents the proportionality coefficient, , represents the variance, , Represents the training parameters of the denoising neural network.
[0030] In one embodiment, the inverse diffusion feature is decoded by using a latent space feature extraction module to obtain a reconstructed ISAR image, including:
[0031] The inverse diffusion features are decoded using the latent space feature extraction module, and the encoded features are input into the decoder to reconstruct the image. The process can be described as
[0032] ;
[0033] in, represents the reconstructed ISAR image, represents the reconstructed optical image, represents the inverse diffusion characteristics corresponding to the ISAR image data, represents the latent space features corresponding to the optical image data, Represents the decoding and reconstruction process.
[0034] In one embodiment, image generation is performed on input optical domain satellite image data according to a trained satellite ISAR image data intelligent generation model to obtain a satellite ISAR image, including:
[0035] The latent space feature extraction module in the trained satellite ISAR image data intelligent generation model is used to extract the latent space features of the input optical domain satellite image data, and then the extracted latent space features are subjected to reverse diffusion denoising to obtain the features after reverse diffusion denoising;
[0036] The inverse diffusion features are input into the latent space feature extraction module for decoding to obtain the reconstructed ISAR image.
[0037] The above-mentioned satellite ISAR image data intelligent generation method based on the diffusion model first constructs a satellite ISAR image data intelligent generation model; the satellite ISAR image data intelligent generation model includes a latent space feature extraction module and an ISAR image generation module based on the diffusion model. The latent space feature extraction module first encodes the image into the latent space through a convolutional neural network, and then models the probability distribution of the latent space features through the Transformer to achieve accurate expression of optical and ISAR images in the latent space. This two-stage method effectively circumvents the limitation that the Transformer cannot process long sequence data, and makes full use of its remote interaction ability and global consistency. Then, the ISAR image generation module based on the diffusion model adopts an unconditional generation strategy, reconstructs the diffusion model based on the Brownian bridge equation, designs a continuous time random process, models the data distribution as a point-to-point correspondence, ensures that the probability distribution of each diffusion step in the forward diffusion process is related to the initial and final states of the diffusion process, and achieves high-precision conversion between optical images and ISAR images. This application realizes the intelligent generation task of optical-ISAR image data of satellite targets based on the diffusion model, which has better results than the classic image generation task; and fully extracts the details and structural features in the ISAR image domain based on VQGAN, and avoids problems such as training instability and mode collapse; at the same time, it combines the Brownian bridge equation with the diffusion model for the first time, and successfully applies the scalability and robustness of the diffusion model to the optical-ISAR image translation task, realizing end-to-end latent space feature conversion. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 A schematic diagram of a flow chart of a method for intelligently generating satellite ISAR image data based on a diffusion model in one embodiment;
[0039] Figure 2 A schematic diagram of a basic flow chart of a latent space feature extraction module in one embodiment;
[0040] Figure 3 A basic structure diagram of a variational autoencoder in one embodiment;
[0041] Figure 4 It is a schematic diagram of an ISAR image generation module based on a diffusion model in another embodiment;
[0042] Figure 5 A flow chart of a model test in one embodiment;
[0043] Figure 6 A schematic diagram of satellite optical-ISAR image translation results in one embodiment;
[0044] Figure 7A schematic diagram of a quantitative evaluation index for satellite optical-ISAR image translation in one embodiment;
[0045] Figure 8 FIG. 1 is a schematic diagram of the accuracy of classifying ISAR images generated by the model using VGG-16 and ResNet-18 in one embodiment; wherein, Figure 8 (a) is a schematic diagram of the classification accuracy of VGG-16. Figure 8 (b) is a schematic diagram of the classification accuracy of ResNet-18;
[0046] Fig. 9 A schematic diagram of an ablation experiment of a pre-trained VQGAN module in one embodiment; DETAILED DESCRIPTION
[0047] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0048] In one embodiment, Figure 1 As shown, a method for intelligently generating satellite ISAR image data based on a diffusion model is provided, comprising the following steps:
[0049] Step 102, acquiring optical-ISAR image data; constructing a satellite ISAR image data intelligent generation model; the satellite ISAR image data intelligent generation model includes a latent space feature extraction module and an ISAR image generation module based on a diffusion model.
[0050] The paired optical-ISAR image data are obtained as the training data set for the satellite ISAR image data intelligent generation model.
[0051] Step 104, in the latent space feature extraction module, the optical-ISAR image data is encoded into the latent space according to the variational autoencoder to obtain a feature vector; two VQGAN models are pre-trained in the optical image domain and the ISAR image domain respectively, and the feature vector is subjected to latent space feature extraction according to the VQGAN model to obtain the latent space features of the optical image domain and the ISAR image domain.
[0052] The latent space feature extraction module uses VQGAN as the basic architecture, combining the advantages of CNN and Transformer. Figure 2 The following is a basic flow chart of this module. First, the distribution characteristics of the hidden variables in the intermediate process are modeled as Gaussian distribution, and the feature representation is changed from a fixed value to a range representation, which can achieve the effect of generating pictures from random vectors. Its basic structure is as follows Figure 3 Input image data The probability distribution of It can be expressed as:
[0053] ;
[0054] In the formula Represents the hidden variables of the middle layer. Combined with the Gaussian distribution calculation formula, the loss function can be expressed as:
[0055] ;
[0056] In the formula, and Respectively represent the corresponding variance and mean in the encoding process, Represents the decoding process of sampling recovery.
[0057] Based on the variational autoencoder, VQGAN introduces a lookup table codebook, which converts the feature vector extracted by the encoder into the closest vector in the codebook. Instead, the continuous features are discretized and the feature information is converted into a sequence of text types, which expands the application scope of the model. , ISAR image domain Two VQGAN models were pre-trained to extract feature encodings of the corresponding image domains, providing accurate feature expressions for subsequent image generation tasks. Latent space features of ISAR and optical images , It can be expressed as:
[0058] ;
[0059] in, , represents the input optical,ISAR image, , Represents the parameters of the network.
[0060] This module also has a decoding module, which is used to decode and reconstruct the features of the ISAR image generation module based on the diffusion model. The process can be described as:
[0061] ;
[0062] in , Represents the reconstructed image.
[0063] At the same time, this application continuously optimizes the latent space feature extraction process by designing the loss function of the latent space feature extraction of optical images and the loss function of the latent space feature extraction of ISAR images, which greatly improves the accuracy of deep feature extraction. The loss function of the latent space feature extraction of optical images can be expressed as:
[0064] ;
[0065] in, represents the encoder, Represents the decoder, represents a fine-tunable weighting parameter. Represents the "stop gradient" operation. The first term of the loss function represents the image reconstruction loss, and the second and third terms guide and The fourth item is used for training Against the generator. Loss function and Similarly, it can be expressed as:
[0066] ;
[0067] According to the loss function, the network parameter update method for the optical image domain and the ISAR image domain can be set as follows:
[0068] parameter In each iteration of the training process, an optimization update is performed, where ,The parameter update in the optical image domain can be expressed as:
[0069] ;
[0070] in, Contains the learning rate of the parameters, represents the optimizer. The parameter update in the ISAR image domain can be expressed as:
[0071] ;
[0072] Step 106, in the ISAR image generation module based on the diffusion model, the data distribution function and iterative process of the forward diffusion process of the latent space features from the ISAR image domain to the optical image domain are designed according to the Brownian bridge diffusion model to complete the latent space feature fusion, and the fused latent space features are subjected to reverse diffusion denoising to obtain the reverse diffusion features.
[0073] The ISAR image generation module based on the diffusion model combines the Brownian bridge model with the diffusion model, models the data distribution as a point-to-point correspondence, and can achieve image translation in the latent space through the diffusion process. Figure 4 The basic process of this module is shown. First, in the forward diffusion process, the Brownian bridge diffusion model defines the starting point of the forward process. and end point Complete the feature fusion in the latent space. Specifically, the state distribution of each diffusion step in the forward process is All of them satisfy the functional relationship with the data distribution of the initial and final states, and this relationship can be modeled as:
[0074] ;
[0075] definition , , ,use Represents the variance, and the iterative process of forward diffusion can be expressed as:
[0076] ;
[0077] The calculation formula for the forward transfer probability can be derived as follows:
[0078] ;
[0079] in, , the forward process establishes a latent space from the image domain I To the image domain O The mapping provides the basic structure for the training framework of the diffusion process.
[0080] Reverse diffusion process using neural network fitting to denoise the process To approximate the transition probability , so that the difference between them is minimized. In addition, the average The learning parameters of the network can be Therefore, the transition probability formula of the reverse process can be modeled as:
[0081] ;
[0082] The reverse diffusion process uses the optimized evidence lower bound (ELBO) to accurately guide the training, which can be proved by mathematical derivation. It can be expressed as , and The linear combination of , and The training objective can be expressed as:
[0083] ;
[0084] During the model training process, the relevant parameters and framework of the forward diffusion process and the reverse diffusion process are designed and trained. During the model testing and application process, the trained reverse process is used to implement latent space optical-ISAR image translation.
[0085] Step 108: input the inverse diffusion features into a latent space feature extraction module for decoding to obtain a reconstructed ISAR image.
[0086] Step 110, using optical-ISAR image data, a pre-set loss function for latent space feature extraction in the optical image domain and a loss function for latent space feature extraction in the ISAR image domain, and taking the optimization of the lower bound of evidence as the objective function of the reverse diffusion process to train the satellite ISAR image data intelligent generation model, to obtain a trained satellite ISAR image data intelligent generation model.
[0087] The model training process of the present application is as follows: (1) taking paired optical-ISAR image data of multiple types and multiple angles of satellite targets as input; (2) pre-training the VQGAN model on the optical and ISAR data sets of the satellite targets in the training set, respectively, to achieve latent space feature extraction and representation, calculate the loss function, and update the two types of VQGAN network parameters respectively; (3) using the ISAR image generation module based on the diffusion model to achieve latent space image translation for the paired images in the training set, calculate the loss function, and update the network parameters with the optimization of the evidence lower bound as the objective function of the inverse diffusion process, thus obtaining a trained satellite ISAR image data intelligent generation model.
[0088] Among them, during the reverse diffusion process training, the network parameters At each iteration it is updated as follows:
[0089] ;
[0090] In addition, a testing process was designed, including: (1) inputting satellite image data in the optical domain; (2) performing latent space feature extraction on optical satellite images through the feature extraction module; (3) performing latent space optical-to-ISAR image translation through the image generation module; (4) using VQGAN to decode the converted latent space features into real ISAR images; and (5) demonstrating the effectiveness and universality of the model through comparative tests and classification experiments. Figure 5 The specific testing process of the model is demonstrated.
[0091] Step 112, performing image generation on the input optical domain satellite image data according to the trained satellite ISAR image data intelligent generation model to obtain a satellite ISAR image.
[0092] In the above-mentioned satellite ISAR image data intelligent generation method based on diffusion model, firstly, a satellite ISAR image data intelligent generation model is constructed; the satellite ISAR image data intelligent generation model includes a latent space feature extraction module and an ISAR image generation module based on diffusion model. The latent space feature extraction module first encodes the image into the latent space through a convolutional neural network, and then models the probability distribution of the latent space features through Transformer to achieve accurate expression of optical and ISAR images in the latent space. This two-stage method effectively circumvents the limitation that Transformer cannot process long sequence data, and makes full use of its remote interaction ability and global consistency. Then, the ISAR image generation module based on diffusion model adopts an unconditional generation strategy, reconstructs the diffusion model based on the Brownian bridge equation, designs a continuous time random process, models the data distribution as a point-to-point correspondence, ensures that the probability distribution of each diffusion step in the forward diffusion process is related to the initial and final states of the diffusion process, and achieves high-precision conversion between optical images and ISAR images. This application realizes the intelligent generation task of optical-ISAR image data of satellite targets based on the diffusion model, which has better results than the classic image generation task; and fully extracts the details and structural features in the ISAR image domain based on VQGAN, and avoids problems such as training instability and mode collapse; at the same time, it combines the Brownian bridge equation with the diffusion model for the first time, and successfully applies the scalability and robustness of the diffusion model to the optical-ISAR image translation task, realizing end-to-end latent space feature conversion.
[0093] In one embodiment, latent space features are extracted from feature vectors according to the VQGAN model to obtain latent space features in the optical image domain and the ISAR image domain, including:
[0094] According to the VQGAN model, the latent space features of the feature vector are extracted, and the latent space features of the optical image domain and the ISAR image domain are obtained as follows:
[0095] ;
[0096] in, , represents the input ISAR image and optical image, , represents the latent space features extracted from ISAR images and the latent space features extracted from optical images, , Represents the parameters of the VQGAN model in the ISAR image domain and the optical image domain.
[0097] In one embodiment, the loss function for latent space feature extraction in the optical image domain is:
[0098] ;
[0099] in, represents the reconstructed optical image, represents the encoder in the optical image domain, represents the decoder in the optical image domain, represents the fine-tunable weighting parameters in the optical image domain, represents the "stop gradient" operation, Representing latent space features of optical image data.
[0100] In one embodiment, the loss function for latent space feature extraction in the ISAR image domain is:
[0101] ;
[0102] in, represents the reconstructed ISAR image, represents the encoder in the ISAR image domain, represents the decoder in the ISAR image domain, represents the fine-tunable weighting parameters in the ISAR image domain, represents the "stop gradient" operation, Represents the latent space characteristics of ISAR image data.
[0103] In one embodiment, the data distribution function of the forward diffusion process is:
[0104] ;
[0105] in, represents the latent space features of optical image data, represents the latent space characteristics of ISAR image data, represents the latent space features of the image corresponding to the diffusion time step t, represents the diffusion time step, Indicates that the distribution function satisfies the normal distribution relationship. represents the total diffusion step length, Represents the identity matrix.
[0106] In one embodiment, the iterative process of the forward diffusion process is:
[0107] ;
[0108] in, , , , Represents variance.
[0109] In one embodiment, the objective function of the reverse diffusion process is
[0110] ;
[0111] in, represents the latent space features of optical image data, represents the latent space characteristics of ISAR image data, represents the latent space features of the image corresponding to the diffusion time step t, represents the proportionality coefficient, , represents the variance, , Represents the training parameters of the denoising neural network.
[0112] In one embodiment, the inverse diffusion feature is decoded by using a latent space feature extraction module to obtain a reconstructed ISAR image, including:
[0113] The inverse diffusion features are decoded using the latent space feature extraction module, and the encoded features are input into the decoder to reconstruct the image. The process can be described as
[0114] ;
[0115] in, represents the reconstructed ISAR image, represents the reconstructed optical image, represents the inverse diffusion characteristics corresponding to the ISAR image data, represents the inverse diffusion feature corresponding to the optical image data, Represents the decoding and reconstruction process.
[0116] In one embodiment, image generation is performed on input optical domain satellite image data according to a trained satellite ISAR image data intelligent generation model to obtain a satellite ISAR image, including:
[0117] The latent space feature extraction module in the trained satellite ISAR image data intelligent generation model is used to extract the latent space features of the input optical domain satellite image data, and then the extracted latent space features are subjected to reverse diffusion denoising to obtain the features after reverse diffusion denoising;
[0118] The inverse diffusion features are input into the latent space feature extraction module for decoding to obtain the reconstructed ISAR image.
[0119] In one embodiment, the present application is compared with the classic image translation model to verify the accuracy and effectiveness of the technology. At the same time, the satellite ISAR images generated by the present application and the real ISAR images are classified using the classic classifiers VGG-16 and ResNet-18, respectively, proving that the satellite ISAR images generated by the technology have high quality. Finally, the necessity of the latent space feature extraction module is proved by the ablation experiment.
[0120] like Figure 6 The schematic diagram of the satellite optical-ISAR image translation results shown includes five methods for image translation, including GyeleGAN (representing the image generation model of the Generative Adversarial Network (GAN).), AGGyeleGAN (representing adaptive guided GyeleGAN", which is a variant of GyeleGAN and includes an adaptive guidance mechanism), UNIT (representing a unified image to image translation network), MUNIT (representing multimodal unsupervised) and the present application. The ISAR images generated by other comparison methods show clear contours, but the images they generate lack detailed features, the structural consistency is not ideal, and the distribution of scattering points in the image is not well preserved; in addition, they fail to capture important internal details. In contrast, the present invention can accurately capture the inherent scattering characteristics of ISAR images, thereby generating ISAR images with better visual effects.
[0121] like Figure 7 As shown in the schematic diagram of quantitative evaluation indicators of satellite optical-ISAR image translation, it can be seen from the analysis of quantitative experimental results that by extracting features in the latent space and combining the Brownian bridge equation with the diffusion model, the quality of the generated image is greatly improved in clarity and similarity compared with the comparative method, proving its feasibility and strong competitiveness in image generation tasks.
[0122] like Figure 8 As shown in the schematic diagram of the accuracy of classifying ISAR images generated by the model using VGG-16 and ResNet-18, Figure 8 The unit is %, and the original data set is expanded using satellite ISAR images generated by five different methods. Each time, 200, 400, 800, and 1200 ISAR images generated by the model are added to the original data set, and then retraining is performed and the classification accuracy is calculated. Among them, Figure 8 (a) is a schematic diagram of the classification accuracy of VGG-16. Figure 8 (b) is a schematic diagram of the classification accuracy of ResNet-18. The experimental results show that this application improves the classification accuracy of the classifier, with the maximum improvement reaching 13%. However, when the generated images of the other four translation models are added to the classifier, the accuracy of both classifiers decreases.
[0123] like Fig. 9 As shown in the schematic diagram of the ablation experiment of the pre-trained VQGAN module, it can be seen from the visual effects of the ablation experiment that the ISAR image obtained by latent space feature extraction has a clearer geometric structure and finer details. The average quantitative indicators LPIPS and IS are significantly inferior to the complete model in the experiment without using VQGAN, which proves that the latent space feature extraction module is conducive to more effective feature fusion in the latent space, thereby improving the comprehensiveness, accuracy and robustness of the feature extraction process and achieving better image translation results.
[0124] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0125] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0126] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A method for intelligently generating satellite ISAR image data based on a diffusion model, characterized in that: The method comprises: Acquire optical-ISAR image data; construct a satellite ISAR image data intelligent generation model; the satellite ISAR image data intelligent generation model includes a latent space feature extraction module and an ISAR image generation module based on a diffusion model; In the latent space feature extraction module, the optical-ISAR image data is encoded into a latent space according to a variational autoencoder to obtain a feature vector; two VQGAN models are pre-trained in the optical image domain and the ISAR image domain respectively, and latent space features are extracted from the feature vector according to the VQGAN model to obtain latent space features in the optical image domain and the ISAR image domain; In the ISAR image generation module based on the diffusion model, a data distribution function and an iterative process of the forward diffusion process of the latent space features from the ISAR image domain to the optical image domain are designed according to the Brownian bridge diffusion model to complete the latent space feature fusion, and the fused latent space features are subjected to reverse diffusion denoising to obtain reverse diffusion features; Inputting the inverse diffusion feature into the latent space feature extraction module for decoding to obtain a reconstructed ISAR image; The satellite ISAR image data intelligent generation model is trained by using the optical-ISAR image data, a pre-set loss function for extracting latent space features in the optical image domain, and a loss function for extracting latent space features in the ISAR image domain, and optimizing the lower bound of evidence as the objective function of the reverse diffusion process to obtain a trained satellite ISAR image data intelligent generation model; Performing image generation on input optical domain satellite image data according to the trained satellite ISAR image data intelligent generation model to obtain a satellite ISAR image; The data distribution function of the forward diffusion process is: in, represents the latent space features of optical image data, represents the latent space characteristics of ISAR image data, represents the latent space features of the image corresponding to the diffusion time step t, represents the diffusion time step, Indicates that the distribution function satisfies the normal distribution relationship. represents the total diffusion step length, represents the identity matrix; The iterative process of the forward diffusion process is in, , , , Represents variance.
2. The method according to claim 1, characterized in that Performing latent space feature extraction on the feature vector according to the VQGAN model to obtain latent space features in the optical image domain and the ISAR image domain, including: According to the VQGAN model, the latent space features of the feature vector are extracted, and the latent space features of the optical image domain and the ISAR image domain are obtained as follows: in, , represents the input ISAR image and optical image, , represents the latent space features extracted from ISAR images and the latent space features extracted from optical images, , Represents the parameters of the VQGAN model in the ISAR image domain and the optical image domain.
3. The method according to claim 2, characterized in that The loss function for latent space feature extraction in the optical image domain is: in, represents the reconstructed optical image, represents the encoder in the optical image domain, represents the decoder in the optical image domain, represents the fine-tunable weighting parameters in the optical image domain, represents the "stop gradient" operation, Representing latent space features of optical image data.
4. The method according to claim 1, characterized in that: The loss function for latent space feature extraction in the ISAR image domain is: in, represents the input ISAR image, represents the reconstructed ISAR image, represents the encoder in the ISAR image domain, represents the decoder in the ISAR image domain, represents the fine-tunable weighting parameters in the ISAR image domain, represents the "stop gradient" operation, Represents the latent space characteristics of ISAR image data.
5. The method according to claim 1, characterized in that The objective function of the reverse diffusion process is: in, represents the latent space features of optical image data, represents the latent space characteristics of ISAR image data, represents the latent space features of the image corresponding to the diffusion time step t, represents the proportionality coefficient, , represents the variance, , Represents the training parameters of the denoising neural network.
6. The method according to claim 1, characterized in that The inverse diffusion feature is decoded by using a latent space feature extraction module to obtain a reconstructed ISAR image, including: The inverse diffusion feature is decoded by using the latent space feature extraction module, and the process of inputting the encoded feature into the decoder to reconstruct the image is described as follows: in, represents the reconstructed ISAR image, represents the reconstructed optical image, represents the inverse diffusion characteristics corresponding to the ISAR image data, represents the inverse diffusion feature corresponding to the optical image data, Represents the decoding and reconstruction process.
7. The method according to claim 1, characterized in that The method generates an image of the satellite image data in the optical domain according to the trained satellite ISAR image data intelligent generation model to obtain a satellite ISAR image, including: The latent space feature extraction module in the trained satellite ISAR image data intelligent generation model is used to extract the latent space features of the input optical domain satellite image data, and then the extracted latent space features are subjected to inverse diffusion denoising to obtain inverse diffusion features; The inverse diffusion feature is input into the latent space feature extraction module for decoding to obtain a reconstructed ISAR image.
