Optical-to-ISAR Image Conversion Method and Device Based on Hierarchical Domain Prior

Abstract

The present application relates to a method and apparatus for optical-ISAR image conversion based on hierarchical domain priors. The method comprises: proposing an optical-radar satellite ISAR image generation model based on hierarchical domain priors, namely a cross-modal conversion network. Based on the cycle consistency theory, the network combines the scattering physical domain priors of ISAR images with the classification domain priors oriented to classification tasks, constructs a new scattering consistency loss and classification consistency loss, optimizes the loss function and iterative framework of the model, and based on the constructed satellite target optical image data training samples, uses CycleGAN as the backbone network, and minimizes the network loss function to enable the model to learn the mapping relationship between the optical image domain and the ISAR image domain, ultimately achieving high-quality and high-training-effectiveness generation of satellite target ISAR data.

Description

Technical Field

[0001] The present application relates to the field of radar image intelligent processing and computer intelligence technology, and in particular to a method and device for optical-to-ISAR image conversion based on hierarchical domain priors. Background Art

[0002] Deep learning has achieved great success in target classification and recognition, but the feasibility and accuracy of this method are limited by the number and quality of training samples. In recent years, with the development of aerospace technology, the number of satellites has exploded, and the classification and recognition of satellite targets has occupied an increasingly important position in the fields of military, communications, and national security. Typically, the training data for satellite target recognition and classification tasks uses inverse synthetic aperture radar (ISAR) images that can be imaged all day and all weather. However, due to factors such as Doppler shift, sidelobe interference, spatial clutter, and military confrontation caused by target motion, the obtained ISAR radar images have poor imaging quality and data collection is extremely difficult, making it difficult to train satellite target recognition and classification networks based on deep learning. Therefore, the performance of satellite target recognition and classification methods based on deep learning is greatly limited; at the same time, the optical images corresponding to satellite targets have high imaging quality and are easy to obtain. Therefore, the present invention is committed to generating ISAR images with scattering physical authenticity and classification task training effectiveness from satellite optical images, providing adaptable training samples for deep learning-based classification models. Summary of the Invention

[0003] Based on this, it is necessary to provide an optical-ISAR image conversion method and device based on hierarchical domain priors that can construct radar ISAR images from satellite optical images to address the above technical problems.

[0004] An optical-to-ISAR image conversion method based on hierarchical domain priors, the method comprising:

[0005] Acquire an image training set, wherein the image training set includes a plurality of ISAR training images and optical training images having the same target category;

[0006] Inputting a group of unpaired ISAR training images and optical training images in the image training set into a cross-modal conversion network to obtain corresponding ISAR reconstructed images and optical reconstructed images, wherein the image input cross-modal conversion network includes two generators and a discriminator corresponding to each generator;

[0007] A total loss function is obtained by calculating according to the ISAR training image, the optical training image, the ISAR reconstructed image, and the optical reconstructed image;

[0008] Extracting the scattering point feature matrices corresponding to the ISAR training image and the ISAR reconstructed image respectively by a scale-invariant feature transformation algorithm, and constructing a scattering feature consistency loss function according to the scattering point feature matrices;

[0009] Extracting classification features corresponding to the ISAR training image and the ISAR reconstructed image based on the trained classification network, and constructing a classification consistency loss function according to the classification features;

[0010] In each iterative training, the generator and the discriminator in the cross-modal conversion network are updated according to the total loss function, the scattering feature consistency loss function, and the classification consistency loss function;

[0011] When accumulating a preset number of iterative trainings, constructing a meta-learning loss function based on a meta-learning strategy according to a preset number of scattering feature consistency loss functions and a classification consistency loss function, and updating the generator in the cross-modal conversion network according to the meta-learning loss function;

[0012] Until the loss functions converge, a trained cross-modal conversion network is obtained;

[0013] An optical target image to be converted is acquired, and the optical target image to be converted is converted using the trained cross-modal conversion network to obtain a corresponding ISAR converted image.

[0014] In one embodiment, the cross-modal conversion network adopts a closed-loop framework structure formed by two generative adversarial networks.

[0015] In one embodiment, inputting a set of unpaired ISAR training images and optical training images in the image training set into a cross-modal conversion network comprises:

[0016] The ISAR training image is mapped from the ISAR image domain to the optical image domain by a first generator and converted into a corresponding pseudo-optical image, and then the pseudo-optical image is mapped from the optical image domain to the ISAR image domain and converted into a corresponding ISAR reconstructed image by a second generator;

[0017] The optical image is mapped from the optical image domain to the ISAR image domain by the second generator to be converted into a pseudo ISAR image, and then the pseudo ISAR image is mapped from the ISAR image domain to the optical image domain by the first generator to be converted into an optically reconstructed image.

[0018] In one embodiment, the total loss function includes an adversarial loss function, a cycle consistency loss function, and a consistency loss function;

[0019] The total loss function is expressed as:

[0020] L CycleGAN =L IO +L OI +λ cyc L cyc +λ identity L identity

[0021] In the above formula, L IO and L OI Denote the adversarial loss functions of the first generator and the second generator, L cyc represents the cycle consistency loss function, L identity represents the consistency loss function, λ cyc and λ identity They represent the proportional weights of the cycle consistency loss function and the consistency loss function respectively.

[0022] In one embodiment, the scattering feature consistency loss function is expressed as:

[0023]

[0024] In the above formula, I represents the ISAR training image, Ps(I) represents the scattering point feature matrix extracted from the ISAR training image by the scale-invariant feature transformation algorithm, and G OI (G IO (I)) represents the ISAR reconstructed image constructed by the first generator and the second generator, Ps(G OI (G IO (I))) represents the scattering point feature matrix extracted from the ISAR reconstructed image through the scale-invariant feature transformation algorithm.

[0025] In one embodiment, the classification consistency loss function is expressed as:

[0026] L CD =(H(l,(R(I)))-H(l,R(G OI (G IO (I))))) 2

[0027] In the above formula, H represents the cross entropy function, l represents the classification label of the ISAR training image, and R() represents the trained classification network.

[0028] In one embodiment, the meta-learning loss function is expressed as:

[0029]

[0030] In the above formula, λ SD and λ CDThey represent the weight parameters of the scattering feature consistency function and the classification feature consistency function respectively, s = 1, 2, ..., M / N represents the meta-learning update step, where M represents the number of training samples and N represents the meta-learning update interval.

[0031] In one embodiment, converting the optical target image to be converted using the trained cross-modal conversion network includes:

[0032] The optical target image to be converted is converted using the second generator in the trained cross-modal conversion network.

[0033] In one embodiment, the targets in the ISAR training image, the optical training image, and the optical target image to be converted are all satellites.

[0034] An optical-to-ISAR image conversion device based on hierarchical domain prior, the device comprising:

[0035] An image training set acquisition module is used to acquire an image training set, wherein the image training set includes a plurality of ISAR training images and optical training images having the same target category;

[0036] a cross-modal conversion network input module, configured to input a set of unpaired ISAR training images and optical training images from the image training set into the cross-modal conversion network to obtain corresponding ISAR reconstructed images and optical reconstructed images, wherein the image input cross-modal conversion network includes two generators and a discriminator corresponding to each generator;

[0037] a total loss function calculation module, configured to calculate a total loss function based on the ISAR training image, the optical training image, the ISAR reconstructed image, and the optical reconstructed image;

[0038] a scattering feature consistency loss function construction module, configured to extract the scattering point feature matrices corresponding to the ISAR training image and the ISAR reconstructed image respectively through a scale-invariant feature transformation algorithm, and construct a scattering feature consistency loss function according to the scattering point feature matrices;

[0039] a classification consistency loss function construction module, configured to extract classification features corresponding to the ISAR training image and the ISAR reconstructed image based on a trained classification network, and construct a classification consistency loss function according to the classification features;

[0040] A generator and discriminator update module, configured to update the generator and discriminator in the cross-modal conversion network according to the total loss function, the scattering feature consistency loss function, and the classification consistency loss function in each iterative training;

[0041] A meta-learning loss function update module is used to construct a meta-learning loss function based on a preset number of scattering feature consistency loss functions and a classification consistency loss function based on a meta-learning strategy when accumulating a preset number of iterative trainings, and update the generator in the cross-modal conversion network according to the meta-learning loss function;

[0042] A trained cross-modal conversion network obtaining module is used until each of the loss functions converges, thereby obtaining a trained cross-modal conversion network;

[0043] The optical-to-radar ISAR image conversion module is used to obtain an optical target image to be converted, and convert the optical target image to be converted using the trained cross-modal conversion network to obtain a corresponding ISAR converted image.

[0044] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0045] Acquire an image training set, wherein the image training set includes a plurality of ISAR training images and optical training images having the same target category;

[0046] Inputting a group of unpaired ISAR training images and optical training images in the image training set into a cross-modal conversion network to obtain corresponding ISAR reconstructed images and optical reconstructed images, wherein the image input cross-modal conversion network includes two generators and a discriminator corresponding to each generator;

[0047] A total loss function is obtained by calculating according to the ISAR training image, the optical training image, the ISAR reconstructed image, and the optical reconstructed image;

[0048] Extracting the scattering point feature matrices corresponding to the ISAR training image and the ISAR reconstructed image respectively by a scale-invariant feature transformation algorithm, and constructing a scattering feature consistency loss function according to the scattering point feature matrices;

[0049] Extracting classification features corresponding to the ISAR training image and the ISAR reconstructed image based on the trained classification network, and constructing a classification consistency loss function according to the classification features;

[0050] In each iterative training, the generator and the discriminator in the cross-modal conversion network are updated according to the total loss function, the scattering feature consistency loss function, and the classification consistency loss function;

[0051] When accumulating a preset number of iterative trainings, constructing a meta-learning loss function based on a meta-learning strategy according to a preset number of scattering feature consistency loss functions and a classification consistency loss function, and updating the generator in the cross-modal conversion network according to the meta-learning loss function;

[0052] Until the loss functions converge, a trained cross-modal conversion network is obtained;

[0053] An optical target image to be converted is acquired, and the optical target image to be converted is converted using the trained cross-modal conversion network to obtain a corresponding ISAR converted image.

[0054] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0055] Acquire an image training set, wherein the image training set includes a plurality of ISAR training images and optical training images having the same target category;

[0056] Inputting a group of unpaired ISAR training images and optical training images in the image training set into a cross-modal conversion network to obtain corresponding ISAR reconstructed images and optical reconstructed images, wherein the image input cross-modal conversion network includes two generators and a discriminator corresponding to each generator;

[0057] A total loss function is obtained by calculating according to the ISAR training image, the optical training image, the ISAR reconstructed image, and the optical reconstructed image;

[0058] Extracting the scattering point feature matrices corresponding to the ISAR training image and the ISAR reconstructed image respectively by a scale-invariant feature transformation algorithm, and constructing a scattering feature consistency loss function according to the scattering point feature matrices;

[0059] Extracting classification features corresponding to the ISAR training image and the ISAR reconstructed image based on the trained classification network, and constructing a classification consistency loss function according to the classification features;

[0060] In each iterative training, the generator and the discriminator in the cross-modal conversion network are updated according to the total loss function, the scattering feature consistency loss function, and the classification consistency loss function;

[0061] When accumulating a preset number of iterative trainings, constructing a meta-learning loss function based on a meta-learning strategy according to a preset number of scattering feature consistency loss functions and a classification consistency loss function, and updating the generator in the cross-modal conversion network according to the meta-learning loss function;

[0062] Until the loss functions converge, a trained cross-modal conversion network is obtained;

[0063] An optical target image to be converted is acquired, and the optical target image to be converted is converted using the trained cross-modal conversion network to obtain a corresponding ISAR converted image.

[0064] The above-mentioned optical-to-ISAR image conversion method and device based on hierarchical domain prior proposes an optical-to-radar satellite ISAR image generation model based on hierarchical domain prior, that is, a cross-modal conversion network. Based on the cycle consistency theory, the network combines the scattering physical domain prior of ISAR images with the classification domain prior oriented by classification tasks to construct a new scattering consistency loss and classification consistency loss, optimize the model's loss function and iterative framework, and also based on the constructed satellite target optical image data training samples, with CycleGAN as the backbone network, by minimizing the network loss function, so that the model learns the mapping relationship between the optical image domain and the ISAR image domain, and finally realizes the generation of high-quality and high-training-effectiveness satellite target ISAR data. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 1 is a flow chart of an optical-to-ISAR image conversion method based on hierarchical domain priors in one embodiment;

[0066] Figure 2 Schematic diagram of the CycleGAN framework in one embodiment;

[0067] Figure 3 Schematic diagram of a framework of a cross-modal conversion network in one embodiment;

[0068] Figure 4 This is a schematic diagram of the satellite target optical-ISAR image conversion results in one of the simulation experiments;

[0069] Figure 5 Schematic diagram of optical-ISAR image conversion in multiple directions in one of the simulation experiments;

[0070] Figure 6 A schematic diagram of the results generated for an unfamiliar satellite target in one of the simulation experiments;

[0071] Figure 7 This is a structural block diagram of the XXX device in one embodiment;

[0072] Figure 8 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0074] like Figure 1 As shown, a method for optical-ISAR image conversion based on hierarchical domain prior is provided, which includes the following steps:

[0075] Step S100, obtaining an image training set, where the image training set includes multiple ISAR training images and optical training images with the same target category;

[0076] Step S110: Input a set of unpaired ISAR training images and optical training images from the image training set into a cross-modal conversion network to obtain corresponding ISAR reconstructed images and optical reconstructed images. The image input cross-modal conversion network includes two generators and a discriminator corresponding to each generator.

[0077] Step S120, calculating a total loss function based on the ISAR training image, the optical training image, the ISAR reconstructed image, and the optical reconstructed image;

[0078] Step S130, extracting the scattering point feature matrices corresponding to the ISAR training image and the ISAR reconstructed image respectively through a scale-invariant feature transformation algorithm, and constructing a scattering feature consistency loss function according to the scattering point feature matrices;

[0079] Step S140, extracting classification features corresponding to the ISAR training image and the ISAR reconstructed image based on the trained classification network, and constructing a classification consistency loss function according to the classification features;

[0080] Step S150: in each iterative training, the generator and the discriminator in the cross-modal conversion network are updated according to the total loss function, the scattering feature consistency loss function, and the classification consistency loss function;

[0081] Step S160, when a preset number of iterative trainings are accumulated, constructing a meta-learning loss function based on the scattering feature consistency loss function and the classification consistency loss function for the preset number of times based on the meta-learning strategy, and updating the generator in the cross-modal conversion network based on the meta-learning loss function;

[0082] Step S170, until all loss functions converge, a trained cross-modal conversion network is obtained;

[0083] Step S180: Acquire the optical target image to be converted, and use the trained cross-modal conversion network to convert the optical target image to obtain a corresponding ISAR conversion image.

[0084] In this embodiment, the target in the cross-modal conversion image can be any target, including aircraft, vehicles, etc., and in one embodiment, the generation of satellite target radar ISAR images, that is, generating ISAR image samples based on the optical image of the target radar for training the deep learning network.

[0085] In this embodiment, based on the cycle consistency theory and combining the scattering physical domain prior of ISAR images with the classification domain prior oriented by the classification task, a new scattering consistency loss and classification consistency loss are constructed to optimize the loss function and iterative framework of the network model.

[0086] In step S100, for generating ISAR sample images for target satellites, the targets in the ISAR training images and optical training images in the image training set are all satellites.

[0087] In step S110, the cross-modal conversion network adopts a closed-loop framework structure formed by two generative adversarial networks (GANs), such as Figure 2 As shown. Where, I and O represent the ISAR domain and the optical domain respectively. IO G is defined as a generator that converts images from the ISAR domain to the optical domain. OI On the contrary, it converts the image from the optical domain to the ISAR domain. Ι , D O Defined as two discriminators to determine whether the input image comes from the generator.

[0088] Specifically, a set of unpaired ISAR training images and optical training images from an image training set are input into a cross-modal conversion network. The steps include: a first generator maps the ISAR training images from the ISAR image domain to the optical image domain and converts them into corresponding pseudo-optical images; a second generator then maps the pseudo-optical images from the optical image domain to the ISAR image domain and converts them into corresponding ISAR reconstructed images; and a second generator maps the optical images from the optical image domain to the ISAR image domain and converts them into pseudo-ISAR images; and the first generator then maps the pseudo-ISAR images from the ISAR image domain to the optical image domain and converts them into optical reconstructed images.

[0089] In step S120, after one iterative training of the cross-modal conversion network, the parameters of the two generators and the discriminator in the network are updated according to the total loss function, where the total loss function includes the adversarial loss function, the cycle consistency loss function and the consistency loss function.

[0090] The total loss function is expressed as:

[0091] L CycleGAN =L IO +L OI +λ cyc L cyc +λ identity L identity (1)

[0092] In formula (1), L IO and L OI Represent the adversarial loss functions of the first generator and the second generator, L cyc represents the cycle consistency loss function, L identity represents the consistency loss function, λ cyc and λ identity They represent the proportional weights of the cycle consistency loss function and the consistency loss function respectively.

[0093] Since the image input to the network should be consistent with the reconstructed image, that is, the image obtained after the input image passes through the two generators, the cycle consistency loss function is expressed as:

[0094] L cyc =E IOI [||G OI (G IO (I))-I||1]+E OIO [||G IO (G OI (O))-O||1] (2)

[0095] In formula (2), IOI represents the ISAR-optical-ISAR image conversion, and OIO represents the optical-ISAR-optical image conversion.

[0096] In order to reduce the loss of color information during the image conversion process, a consistency loss function is added, which is expressed as:

[0097] L identity =E IO [||G IO (I)-I||1]+E OI [||G OI (O)-O||1] (3)

[0098] In step S130, to enhance the fidelity of satellite ISAR image generation, physical domain prior information is introduced. By constructing a scattering feature consistency loss, the scattering features in the ISAR image are associated with the network's optimization updates. During iterations, the network learns the distribution characteristics of scatter points in the ISAR image by minimizing the scattering feature consistency loss, ultimately resulting in an image generation network capable of generating images with realistic ISAR scatter point distribution characteristics.

[0099] Specifically, for the input satellite ISAR image I, the feature vector v of the kth scattering point is extracted k It can be expressed as:

[0100] v k =[c k ,θ k ,M k ] T ,(k=1,2,...,K) (4)

[0101] In formula (4), c k represents the exact coordinates of the extracted scattering points, θ k Indicates the direction of change of the scattering intensity at the scattering point, M k represents θ k The above feature information is obtained by SIFT projection. The entire SIFT projection process is defined as Ps(·). After the input satellite ISAR image is SIFT-projected, a feature matrix consisting of K eigenvectors is obtained. The entire process can be described as:

[0102] [v1,v2,...,v K ] T =Ps(I) (5)

[0103] In formula (5), [v1,v2,...,v K ] T Represents the feature matrix composed of the K extracted eigenvectors. Therefore, after SIFT projection, the scattering physical characteristics in the ISAR physical domain prior are extracted and participate in the subsequent network iterative optimization in the form of a feature matrix, realizing the introduction of ISAR physical domain prior information.

[0104] Specifically, based on the theoretical principle of cycle consistency, in this application, it is believed that the input satellite ISAR image and the satellite ISAR image reconstructed by the network should have the same scattering physical properties, that is, after SIFT projection, the reconstructed satellite ISAR image should have the same feature matrix as the input satellite ISAR image. However, due to insufficient network training and insufficient sampling rate of satellite ISAR image distribution, the scattering feature matrix of the reconstructed satellite ISAR image deviates from the scattering feature matrix of the input satellite ISAR image. Therefore, based on this starting point, a scattering feature consistency loss is constructed in order to enable the network to learn the physical properties of satellite ISAR images. By minimizing this loss function, the model learns the mapping relationship between the scattering physical properties in the ISAR image domain and the geometric structure in the optical image. The scattering feature consistency loss function can be expressed as:

[0105]

[0106] In formula (6), I represents the ISAR training image, Ps(I) represents the scattering point feature matrix extracted from the ISAR training image by the scale-invariant feature transformation algorithm, and G OI (G IO (I)) represents the ISAR reconstructed image constructed by the first generator and the second generator, Ps(G OI (G IO (I))) represents the scattering point feature matrix extracted from the ISAR reconstructed image through the scale-invariant feature transformation algorithm.

[0107] In step 140, to improve the training validity of the generated ISAR data—that is, to ensure that the generated ISAR data can be used to train subsequent recognition and classification networks—a classification network is introduced to project the prior features of the satellite ISAR image classification domain to extract the classification features of the satellite ISAR image. Based on the principle of cycle consistency, a classification consistency loss is constructed. This loss is optimized to enable the network to learn the classification features of the ISAR image, ultimately generating satellite ISAR images with high training validity.

[0108] Specifically, the input satellite ISAR image is classified and projected using a pre-trained classification network. Based on the consistency principle, it is believed that the input satellite ISAR image and the satellite ISAR image reconstructed by the network should have the same classification features. However, due to factors such as insufficient network training, the classification features of the reconstructed satellite ISAR image are often different from those of the input satellite ISAR image. Therefore, the present invention constructs a classification consistency loss function, whose mathematical expression is:

[0109] L CD =(H(l,(R(I)))-H(l,R(G OI (GIO (I))))) 2 (7)

[0110] In formula (7), H represents the cross entropy function, l represents the classification label of the ISAR training image, and R() represents the trained classification network. The output value of the cross entropy function is used to measure the classification performance of the input ISAR image and reflects the prior characteristics of the classification domain of the ISAR image.

[0111] In practical engineering applications, the limited amount of satellite ISAR imagery available for training this application necessitates targeted optimization and improvement of the network framework to address this data shortage. A meta-learning-based optimization strategy, by calculating the average of the scattering feature consistency and classification feature consistency losses, enables the model to learn the shared physical scattering and classification features in satellite ISAR images, improving the model's learning and generative capabilities even with small sample training data.

[0112] Specifically, the meta-learning loss function based on the meta-learning strategy is expressed as:

[0113]

[0114] In formula (8), λ SD and λ CD They represent the weight parameters of the scattering feature consistency function and the classification feature consistency function respectively, s = 1, 2, ..., M / N represents the meta-learning update step, where M represents the number of training samples and N represents the meta-learning update interval.

[0115] Furthermore, to improve the effectiveness of training generated images, the meta-learning update is only for the generator G ΙΟ ,G OI , if and only if s=1,2,...,M / N. Based on the Adam optimizer, the update of the generator network parameters is:

[0116]

[0117]

[0118] In formulas (9) and (10), as well as represents the parameters of the two generators, γ G Represents the learning rate of network parameters.

[0119] like Figure 3 As shown, the cross-modal conversion network includes four network models, namely the generator G IO , G OI , discriminator D I 、D OThe model uses CyleGAN as the backbone framework. Based on the original CycleGAN total loss function, it proposes new scattering feature consistency loss and classification feature consistency loss, and improves the model's parameter update process based on the meta-learning optimization strategy. Specifically, in each iterative update, the model will update the generator and discriminator parameters based on the total loss function of the CycleGAN framework ( Figure 3 where Θ represents the set of generator and discriminator parameters); after D iterations, the model will optimize and update the generator parameters again based on the accumulated meta-learning loss ( Figure 3 where Φ represents the set of generator parameters); ultimately, through continuous iterative optimization, the model will be able to generate satellite ISAR images with high authenticity and high training effectiveness under the condition of a small number of samples.

[0120] In this embodiment, steps S100 to S170 are the process of training the cross-modal conversion network, and the specific process can also be summarized as follows: (1) The cross-modal conversion network obtains unpaired optical and ISAR images from the satellite dataset; (2) In the optical-ISAR-optical conversion, the input optical image is passed through the two generators to obtain a reconstructed optical image, and the adversarial loss and cycle consistency loss based on CycleGAN are calculated at this time; (3) In the ISAR-optical-ISAR conversion, the input ISAR image is passed through the two generators to obtain a reconstructed ISAR image, and the adversarial loss, cycle consistency, scattering feature consistency and classification feature consistency losses are calculated at this time; (4) In each iteration, the total loss function of CycleGAN is calculated, and the generator and discriminator are updated; (5) After D cumulative iterations, the meta-learning loss is calculated, and the generator is updated.

[0121] After training the cross-modal conversion network, the network will be tested. The testing process includes: (1) the model obtains optical images from the satellite dataset; (2) the trained generator G is used to generate the image. OI , convert the input optical image into the corresponding ISAR image; (3) measure the quality of the ISAR image generated by the model by qualitative and quantitative comparison; (4) form a new dataset of the generated ISAR image and the real ISAR image, and use the dataset to train the recognition and classification network to observe whether the network recognition and classification performance is improved.

[0122] In step S180, when the optical target image to be converted is converted using the trained cross-modal conversion network, in practice, the optical target image to be converted is converted using the second generator in the trained cross-modal conversion network.

[0123] In this paper, simulation experiments are also carried out based on this method. Figure 4 As shown in Figure 2, the conversion result of optical to ISAR image based on this method is performed. The results show that the ISAR image generated by this method does have the least scattering points. For example, the scattering point distribution of the satellite solar panel part and the satellite main body is most completely preserved.

[0124] like Figure 5 As shown in Figure 2, in the case of multi-azimuth conversion, the ISAR image generated by this method is clearer and more complete than that of the existing method, and the distribution of scattering points changes with the azimuth of the satellite target.

[0125] When targeting unfamiliar samples, Figure 6 As shown in the figure, in the Inception Score (IS) measurement standard, this method is 0.35 higher than the classic CycleGAN and 0.46 higher than the UNIT method; in the structural similarity index, this method still achieves the highest similarity, which is 0.0339 and 0.2046 higher than CycleGAN and UNIT respectively.

[0126] As shown in Table 1, when the data generated by this method is added to the classification and recognition network, the network performance is improved by 5-8% compared with the original one, and can continue to improve as the amount of generated data increases. In contrast, the data generated by other methods has no significant effect on network performance and even hinders the improvement of network performance.

[0127] Table 1: Training effectiveness experimental results

[0128]

[0129] In the above-mentioned optical-to-ISAR image conversion method based on hierarchical domain prior, an optical-to-radar satellite ISAR image generation model based on hierarchical domain prior is constructed. Based on the cycle consistency theory, the model combines the scattering physical domain prior of ISAR images with the classification domain prior oriented by the classification task, constructs a new scattering consistency loss and classification consistency loss, and optimizes the loss function and iterative framework of the model. Based on the constructed satellite target optical image data training samples, the invention uses CycleGAN as the backbone network, and by minimizing the network loss function, enables the model to learn the mapping relationship between the optical image domain and the ISAR image domain, and finally realizes the generation of high-quality and high-training-effectiveness satellite target ISAR data. A large number of comparative experiments and training effectiveness experiments have proved that the satellite ISAR data generated by the present invention has a real scattering point distribution and can effectively improve the classification and recognition network performance. Furthermore, the present application proposes a physical domain projection of ISAR images based on the scale-invariant feature transform (SIFT) to extract the distribution characteristics of scattering points in satellite ISAR images and obtain the scattering feature consistency loss; and proposes a classification domain projection based on a classification network to extract the classification features of satellite ISAR images and obtain the classification consistency loss. In addition, in order to enable the network to learn the common scattering distribution characteristics and classification features in satellite ISAR images, the present invention builds a hierarchical optimization framework based on the meta-learning optimization strategy, improves the network parameter optimization strategy, and reduces the model's dependence on the amount of sample data. The optical-satellite ISAR radar image data cross-modal generation model based on the hierarchical domain prior in this method can provide an end-to-end model that integrates target data generation and recognition, and generates satellite ISAR images with high authenticity and high training effectiveness. The satellite ISAR images reconstructed from the optical images can effectively improve the performance of satellite classification and recognition algorithms, and can effectively solve the problem of lack of special target training data in military tasks.

[0130] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0131] In one embodiment, Figure 7 As shown, an optical-to-ISAR image conversion device based on hierarchical domain prior is provided, comprising: an image training set acquisition module 200, a cross-modal conversion network input module 210, a total loss function calculation module 220, a scattering feature consistency loss function construction module 230, a classification consistency loss function construction module 240, a generator and discriminator update module 250, a meta-learning-based loss function update module 260, a trained cross-modal conversion network acquisition module 270, and an optical-to-radar ISAR image conversion module 280, wherein:

[0132] An image training set acquisition module 200 is configured to acquire an image training set, wherein the image training set includes a plurality of ISAR training images and optical training images having the same target category;

[0133] A cross-modal conversion network input module 210 is configured to input a set of unpaired ISAR training images and optical training images from the image training set into a cross-modal conversion network to obtain corresponding ISAR reconstructed images and optical reconstructed images. The image input cross-modal conversion network includes two generators and a discriminator corresponding to each generator.

[0134] a total loss function calculation module 220, configured to calculate a total loss function based on the ISAR training image, the optical training image, the ISAR reconstructed image, and the optical reconstructed image;

[0135] A scattering feature consistency loss function construction module 230 is configured to extract scattering point feature matrices corresponding to the ISAR training image and the ISAR reconstructed image respectively through a scale-invariant feature transformation algorithm, and to construct a scattering feature consistency loss function based on the scattering point feature matrices.

[0136] A classification consistency loss function construction module 240 is used to extract classification features corresponding to the ISAR training image and the ISAR reconstructed image based on the trained classification network, and construct a classification consistency loss function according to the classification features;

[0137] A generator and discriminator update module 250 is configured to update the generator and discriminator in the cross-modal conversion network according to the total loss function, the scattering feature consistency loss function, and the classification consistency loss function in each iterative training;

[0138] A meta-learning loss function updating module 260 is configured to construct a meta-learning loss function based on a preset number of scattering feature consistency loss functions and a classification consistency loss function when accumulating a preset number of iterative trainings, and to update the generator in the cross-modal conversion network according to the meta-learning loss function;

[0139] A trained cross-modal conversion network obtaining module 270 is configured to obtain a trained cross-modal conversion network until the loss functions converge.

[0140] The optical-to-radar ISAR image conversion module 280 is used to obtain an optical target image to be converted, and convert the optical target image to be converted using the trained cross-modal conversion network to obtain a corresponding ISAR converted image.

[0141] The specific limitations of the optical-to-ISAR image conversion device based on hierarchical domain priors can be found in the limitations of the optical-to-ISAR image conversion method based on hierarchical domain priors described above and will not be repeated here. Each module in the aforementioned optical-to-ISAR image conversion device based on hierarchical domain priors can be implemented in whole or in part via software, hardware, or a combination thereof. Each of the aforementioned modules can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in a computer device in software form, so that the processor can call and execute the corresponding operations of each of the aforementioned modules.

[0142] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, an optical-to-ISAR image conversion method based on hierarchical domain prior is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0143] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0144] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0145] Acquire an image training set, wherein the image training set includes a plurality of ISAR training images and optical training images having the same target category;

[0146] Inputting a group of unpaired ISAR training images and optical training images in the image training set into a cross-modal conversion network to obtain corresponding ISAR reconstructed images and optical reconstructed images, wherein the image input cross-modal conversion network includes two generators and a discriminator corresponding to each generator;

[0147] A total loss function is obtained by calculating according to the ISAR training image, the optical training image, the ISAR reconstructed image, and the optical reconstructed image;

[0148] Extracting the scattering point feature matrices corresponding to the ISAR training image and the ISAR reconstructed image respectively by a scale-invariant feature transformation algorithm, and constructing a scattering feature consistency loss function according to the scattering point feature matrices;

[0149] Extracting classification features corresponding to the ISAR training image and the ISAR reconstructed image based on the trained classification network, and constructing a classification consistency loss function according to the classification features;

[0150] In each iterative training, the generator and the discriminator in the cross-modal conversion network are updated according to the total loss function, the scattering feature consistency loss function, and the classification consistency loss function;

[0151] When accumulating a preset number of iterative trainings, constructing a meta-learning loss function based on a meta-learning strategy according to a preset number of scattering feature consistency loss functions and a classification consistency loss function, and updating the generator in the cross-modal conversion network according to the meta-learning loss function;

[0152] Until the loss functions converge, a trained cross-modal conversion network is obtained;

[0153] An optical target image to be converted is acquired, and the optical target image to be converted is converted using the trained cross-modal conversion network to obtain a corresponding ISAR converted image.

[0154] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0155] Acquire an image training set, wherein the image training set includes a plurality of ISAR training images and optical training images having the same target category;

[0156] Inputting a group of unpaired ISAR training images and optical training images in the image training set into a cross-modal conversion network to obtain corresponding ISAR reconstructed images and optical reconstructed images, wherein the image input cross-modal conversion network includes two generators and a discriminator corresponding to each generator;

[0157] A total loss function is obtained by calculating according to the ISAR training image, the optical training image, the ISAR reconstructed image, and the optical reconstructed image;

[0158] Extracting the scattering point feature matrices corresponding to the ISAR training image and the ISAR reconstructed image respectively by a scale-invariant feature transformation algorithm, and constructing a scattering feature consistency loss function according to the scattering point feature matrices;

[0159] Extracting classification features corresponding to the ISAR training image and the ISAR reconstructed image based on the trained classification network, and constructing a classification consistency loss function according to the classification features;

[0160] In each iterative training, the generator and the discriminator in the cross-modal conversion network are updated according to the total loss function, the scattering feature consistency loss function, and the classification consistency loss function;

[0161] When accumulating a preset number of iterative trainings, constructing a meta-learning loss function based on a meta-learning strategy according to a preset number of scattering feature consistency loss functions and a classification consistency loss function, and updating the generator in the cross-modal conversion network according to the meta-learning loss function;

[0162] Until the loss functions converge, a trained cross-modal conversion network is obtained;

[0163] An optical target image to be converted is acquired, and the optical target image to be converted is converted using the trained cross-modal conversion network to obtain a corresponding ISAR converted image.

[0164] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0165] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0166] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. An optical-to-ISAR image conversion method based on hierarchical domain priors, characterized by: The method comprises: Acquire an image training set, wherein the image training set includes a plurality of ISAR training images and optical training images having the same target category; Inputting a group of unpaired ISAR training images and optical training images in the image training set into a cross-modal conversion network to obtain corresponding ISAR reconstructed images and optical reconstructed images, wherein the image input cross-modal conversion network includes two generators and a discriminator corresponding to each generator; The total loss function including the adversarial loss function, the cycle consistency loss function and the consistency loss function is calculated based on the ISAR training image, the optical training image, the ISAR reconstructed image and the optical reconstructed image, and is expressed as: In the above formula, as well as Represent the adversarial loss functions of the first generator and the second generator respectively, represents the cycle consistency loss function, represents the consistency loss function, as well as Represent the proportional weights of the cycle consistency loss function and the consistency loss function respectively; The scattering point feature matrices corresponding to the ISAR training image and the ISAR reconstructed image are extracted respectively by the scale-invariant feature transformation algorithm, and a scattering feature consistency loss function is constructed according to the scattering point feature matrix, which is expressed as: In the above formula, represents the ISAR training image, It represents the scattering point feature matrix extracted from the ISAR training image by the scale-invariant feature transformation algorithm. represents the ISAR reconstructed image constructed by the first generator and the second generator, It represents the scattering point feature matrix extracted from the ISAR reconstructed image by the scale-invariant feature transformation algorithm; Extracting classification features corresponding to the ISAR training image and the ISAR reconstructed image based on the trained classification network, and constructing a classification consistency loss function according to the classification features; In each iterative training, the generator and the discriminator in the cross-modal conversion network are updated according to the total loss function, the scattering feature consistency loss function, and the classification consistency loss function; When accumulating a preset number of iterative trainings, a meta-learning loss function is constructed based on the meta-learning strategy according to the preset number of scattering feature consistency loss functions and the classification consistency loss functions, and the generator in the cross-modal conversion network is updated according to the meta-learning loss function, wherein the meta-learning loss function is expressed as: In the above formula, as well as Represent the weight parameters of the scattering feature consistency function and the classification feature consistency function, represents the meta-learning update step, where represents the number of training samples, represents the meta-learning update interval; Until the loss functions converge, a trained cross-modal conversion network is obtained; An optical target image to be converted is acquired, and the optical target image to be converted is converted using the trained cross-modal conversion network to obtain a corresponding ISAR converted image.

2. The optical-to-ISAR image conversion method according to claim 1, wherein: The cross-modal conversion network adopts a closed-loop framework structure formed by two generative adversarial networks.

3. The optical-ISAR image conversion method according to claim 2, characterized in that: Inputting a group of unpaired ISAR training images and optical training images in the image training set into the cross-modal conversion network comprises: The ISAR training image is mapped from the ISAR image domain to the optical image domain by a first generator and converted into a corresponding pseudo-optical image, and then the pseudo-optical image is mapped from the optical image domain to the ISAR image domain and converted into a corresponding ISAR reconstructed image by a second generator; The optical image is mapped from the optical image domain to the ISAR image domain by the second generator to be converted into a pseudo ISAR image, and then the pseudo ISAR image is mapped from the ISAR image domain to the optical image domain by the first generator to be converted into an optically reconstructed image.

4. The optical-to-ISAR image conversion method according to claim 3, characterized in that: The classification consistency loss function is expressed as: In the above formula, represents the cross entropy function, represents the classification label of the ISAR training image, Represents the trained classification network.

5. The optical-ISAR image conversion method according to claim 4, characterized in that: The converting the optical target image to be converted by using the trained cross-modal conversion network includes: The optical target image to be converted is converted using the second generator in the trained cross-modal conversion network.

6. The optical-to-ISAR image conversion method according to any one of claims 1 to 5, characterized in that: The targets in the ISAR training image, the optical training image and the optical target image to be converted are all satellites.

7. An optical-to-ISAR image conversion device based on hierarchical domain priors, characterized in that: The optical-to-ISAR image conversion method based on hierarchical domain prior according to claim 1 is implemented in the device, and the device specifically comprises: An image training set acquisition module is used to acquire an image training set, wherein the image training set includes a plurality of ISAR training images and optical training images having the same target category; a cross-modal conversion network input module, configured to input a set of unpaired ISAR training images and optical training images from the image training set into the cross-modal conversion network to obtain corresponding ISAR reconstructed images and optical reconstructed images, wherein the image input cross-modal conversion network includes two generators and a discriminator corresponding to each generator; a total loss function calculation module, configured to calculate a total loss function based on the ISAR training image, the optical training image, the ISAR reconstructed image, and the optical reconstructed image; a scattering feature consistency loss function construction module, configured to extract the scattering point feature matrices corresponding to the ISAR training image and the ISAR reconstructed image respectively through a scale-invariant feature transformation algorithm, and construct a scattering feature consistency loss function according to the scattering point feature matrices; a classification consistency loss function construction module, configured to extract classification features corresponding to the ISAR training image and the ISAR reconstructed image based on a trained classification network, and construct a classification consistency loss function according to the classification features; A generator and discriminator update module, configured to update the generator and discriminator in the cross-modal conversion network according to the total loss function, the scattering feature consistency loss function, and the classification consistency loss function in each iterative training; A meta-learning loss function update module is used to construct a meta-learning loss function based on a preset number of scattering feature consistency loss functions and a classification consistency loss function based on a meta-learning strategy when accumulating a preset number of iterative trainings, and update the generator in the cross-modal conversion network according to the meta-learning loss function; A trained cross-modal conversion network obtaining module is used until each of the loss functions converges, thereby obtaining a trained cross-modal conversion network; The optical-to-radar ISAR image conversion module is used to obtain an optical target image to be converted, and convert the optical target image to be converted using the trained cross-modal conversion network to obtain a corresponding ISAR converted image.

Images