A Generative Network-Based Optical-Radar ISAR Image Conversion Method and Apparatus

Abstract

This application relates to a method and apparatus for optical-radar ISAR image conversion based on generative networks. The method includes: analyzing and modeling the physical characteristics of the target in the ISAR image, and using a deep learning network as the main technical means to incorporate the distribution characteristics of ISAR scattering points into the loss function of the network training, ultimately converting the optical image of the target into the corresponding ISAR image. The constraint that the scattering point position matrices extracted from the original ISAR image and the reconstructed ISAR image are equal is added to the loss function, enabling the deep learning network to generate more realistic ISAR images.

Description

Technical Field

[0001] This application relates to the fields of intelligent radar image processing and computer intelligence technology, and in particular to an optical-radar ISAR image conversion method and apparatus based on generative networks. Background Technology

[0002] Image translation technology is a major research direction in artificial intelligence, used to solve the style representation of task targets in different image domains. This technology learns from data in different image domains, uncovers their inherent data distribution characteristics, and then uses a generative model to obtain the representation of the task target in the target domain from the original image domain. The mainstream approach in current image translation technology is to train generative adversarial networks (GANs) to obtain generative models for data conversion and generation. Currently, there are many applications of GAN-based image translation technology in the field of airborne remote sensing, mainly focusing on the matching of optical-synthetic aperture (SAR) images and the conversion and generation of SAR images to optical images.

[0003] In existing technologies, although satellites can monitor a designated scene in real time, the monitoring time for each satellite is often only a few minutes because satellites need to orbit. At the same time, although many satellites with overlapping orbits may pass through the same area, each satellite uses different monitoring devices. Therefore, a method that can be generated across domains and modes is needed to uniformly represent the monitoring data of each satellite. Summary of the Invention

[0004] Therefore, it is necessary to provide a generative network-based optical-radar ISAR image conversion method and apparatus that can achieve image domain unification, addressing the aforementioned technical problems.

[0005] An optical-radar ISAR image conversion method based on generative networks, the method comprising:

[0006] Obtain an image training set, which includes multiple ISAR original images and optical images with the same target category;

[0007] The original ISAR image and the optical image in the image training set are input into the generation network. The original ISAR image is mapped from the ISAR image domain to the optical image domain by the first generator in the generation network and transformed into a corresponding pseudo-optical image. Then, the pseudo-optical image is mapped from the optical image domain to the ISAR image domain by the second generator and transformed into a corresponding ISAR reconstructed image. At the same time, the optical image is transformed into a pseudo-ISAR image by the second generator and then the pseudo-ISAR image is transformed into an optical reconstructed image by the first generator.

[0008] Scattering points are extracted from the original ISAR image and the corresponding reconstructed ISAR image to obtain the first scattering point position matrix of the original ISAR image and the second scattering point position matrix of the reconstructed ISAR image.

[0009] The first scattering point position matrix and the second scattering point position matrix of the ISAR reconstructed image are made equal as a constraint condition. Based on the augmented Lagrange operator, the constraint condition is added as a penalty function term to the original loss function to reconstruct the final loss function.

[0010] The generator network is trained based on the final loss function to obtain a trained generator network;

[0011] An optical target image to be converted is acquired, and the second generator in the trained generation network is used to convert the optical target image to obtain the corresponding ISAR converted image.

[0012] In one embodiment, when extracting scattering points from the original ISAR image and the corresponding reconstructed ISAR image, the scattering points of the ISAR image are extracted using the maximum inter-class variance method and the network method.

[0013] In one embodiment, the extraction of scattering points from the ISAR image based on the maximum inter-class variance method and the network method includes:

[0014] The grayscale threshold of the ISAR image is calculated based on the maximum inter-class variance method, and the foreground and background separation is obtained by segmenting the ISAR image based on the grayscale threshold.

[0015] Then, the scattering points of the ISAR image are extracted using the network method and grayscale threshold.

[0016] In one embodiment, the extraction of scattering points from the ISAR image based on the network method and a grayscale threshold includes:

[0017] The ISAR image is divided into multiple networks of the same size;

[0018] Pixels with amplitude values ​​greater than the grayscale threshold are extracted from each network as scattering points, and the corresponding coordinates and amplitude values ​​of each scattering point are recorded.

[0019] The scattering point position matrix is ​​obtained by reconstructing the corresponding coordinates of each scattering point according to the original coordinate order.

[0020] In one embodiment, the generating network adopts a CycleGAN network and further includes a discriminator in the ISAR image domain and a discriminator in the optical image domain. The first generator and the second generator are implemented using a CNN structure, and the two discriminators are implemented using a PatchGAN structure.

[0021] In one embodiment, the original loss function is:

[0022]

[0023] In one embodiment, the final loss function is:

[0024]

[0025] In the above formula, λ scatter Represents the Lagrange multiplier, ρ scatter This represents the coefficient of the penalty function term.

[0026] In one embodiment, training the generator network based on the final loss function to obtain the trained generator network further includes:

[0027] The gradient of the final loss function is calculated, and the parameters of the first generator and the second generator in the generator network are corrected according to the direction of the calculation result until convergence, thus obtaining the trained generator network.

[0028] An optical-radar ISAR image conversion device based on a generative network, the device comprising:

[0029] The training set acquisition module is used to acquire an image training set, which includes multiple ISAR original images and optical images with the same target category.

[0030] The image domain conversion module is used to input the original ISAR image and the optical image from the image training set into the generation network. The original ISAR image is mapped from the ISAR image domain to the optical image domain by the first generator in the generation network and transformed into a corresponding pseudo-optical image. Then, the pseudo-optical image is mapped from the optical image domain to the ISAR image domain by the second generation device and transformed into a corresponding ISAR reconstructed image. At the same time, the optical image is transformed into a pseudo-ISAR image by the second generator and then the pseudo-ISAR image is transformed into an optical reconstructed image by the first generation device.

[0031] The scattering point position matrix reconstruction module is used to extract the scattering points of the original ISAR image and the corresponding reconstructed ISAR image, respectively, to obtain the first scattering point position matrix of the original ISAR image and the second scattering point position matrix of the reconstructed ISAR image.

[0032] The loss function reconstruction module is used to take the equality of the first scattering point position matrix and the second scattering point position matrix of the ISAR reconstructed image as a constraint condition, and add the constraint condition as a penalty function term to the original loss function based on the form of augmented Lagrange operator to reconstruct the final loss function.

[0033] The network training module is used to train the generator network according to the final loss function to obtain the trained generator network.

[0034] The optical-radar ISAR image conversion module is used to acquire the optical target image to be converted, and to convert the optical target image to be converted using the second generator in the trained generation network to obtain the corresponding ISAR converted image.

[0035] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0036] Obtain an image training set, which includes multiple ISAR original images and optical images with the same target category;

[0037] The original ISAR image and the optical image in the image training set are input into the generation network. The original ISAR image is mapped from the ISAR image domain to the optical image domain by the first generator in the generation network and transformed into a corresponding pseudo-optical image. Then, the pseudo-optical image is mapped from the optical image domain to the ISAR image domain by the second generator and transformed into a corresponding ISAR reconstructed image. At the same time, the optical image is transformed into a pseudo-ISAR image by the second generator and then the pseudo-ISAR image is transformed into an optical reconstructed image by the first generator.

[0038] Scattering points are extracted from the original ISAR image and the corresponding reconstructed ISAR image to obtain the first scattering point position matrix of the original ISAR image and the second scattering point position matrix of the reconstructed ISAR image.

[0039] The first scattering point position matrix and the second scattering point position matrix of the ISAR reconstructed image are made equal as a constraint condition. Based on the augmented Lagrange operator, the constraint condition is added as a penalty function term to the original loss function to reconstruct the final loss function.

[0040] The generator network is trained based on the final loss function to obtain a trained generator network;

[0041] An optical target image to be converted is acquired, and the second generator in the trained generation network is used to convert the optical target image to obtain the corresponding ISAR converted image.

[0042] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0043] Obtain an image training set, which includes multiple ISAR original images and optical images with the same target category;

[0044] The original ISAR image and the optical image in the image training set are input into the generation network. The original ISAR image is mapped from the ISAR image domain to the optical image domain by the first generator in the generation network and transformed into a corresponding pseudo-optical image. Then, the pseudo-optical image is mapped from the optical image domain to the ISAR image domain by the second generator and transformed into a corresponding ISAR reconstructed image. At the same time, the optical image is transformed into a pseudo-ISAR image by the second generator and then the pseudo-ISAR image is transformed into an optical reconstructed image by the first generator.

[0045] Scattering points are extracted from the original ISAR image and the corresponding reconstructed ISAR image to obtain the first scattering point position matrix of the original ISAR image and the second scattering point position matrix of the reconstructed ISAR image.

[0046] The first scattering point position matrix and the second scattering point position matrix of the ISAR reconstructed image are made equal as a constraint condition. Based on the augmented Lagrange operator, the constraint condition is added as a penalty function term to the original loss function to reconstruct the final loss function.

[0047] The generator network is trained based on the final loss function to obtain a trained generator network;

[0048] An optical target image to be converted is acquired, and the second generator in the trained generation network is used to convert the optical target image to obtain the corresponding ISAR converted image.

[0049] The aforementioned optical-radar ISAR image conversion method and apparatus based on generative networks, through the analysis and modeling of the target's physical characteristics in ISAR images, uses deep learning networks as the main technical means and incorporates the distribution characteristics of ISAR scattering points into the loss function of network training, ultimately converting the target's optical image into the corresponding ISAR image. The constraint that the scattering point position matrices extracted from the original ISAR image and the reconstructed ISAR image are equal is added to the loss function, enabling the deep learning network to generate more realistic ISAR images. Attached Figure Description

[0050] Figure 1 This is a flowchart illustrating an optical-radar ISAR image conversion method based on a generative network in one embodiment.

[0051] Figure 2 This is a flowchart illustrating an algorithm based on a generative network in one embodiment;

[0052] Figure 3 This is a schematic diagram of the results of ISAR image scattering point extraction in one embodiment;

[0053] Figure 4 This is a schematic diagram illustrating the convergence of the loss function during the training process of a generator network in one embodiment.

[0054] Figure 5 This is a schematic diagram of the optical-ISAR image conversion result using this method in one embodiment;

[0055] Figure 6 This is a schematic diagram of the optical-ISAR image conversion results under multiple directions using this method in one embodiment;

[0056] Figure 7 This is a schematic diagram of the result of optical-ISAR image conversion of an unfamiliar satellite target using the method described in one embodiment;

[0057] Figure 8 This is a block diagram of an optical-radar ISAR image conversion device based on a generative network in one embodiment.

[0058] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0060] like Figure 1 As shown, to address the problem of unified representation of cross-modal data in existing technologies, a generative network-based optical-radar ISAR image conversion method is provided, including the following steps:

[0061] Step S100: Obtain an image training set, which includes multiple ISAR original images and optical images with the same target category;

[0062] Step S110: Input the original ISAR image and the optical image from the image training set into the generation network. The original ISAR image is mapped from the ISAR image domain to the optical image domain by the first generator in the generation network and transformed into the corresponding pseudo-optical image. Then, the pseudo-optical image is mapped from the optical image domain to the ISAR image domain by the second generator and transformed into the corresponding ISAR reconstructed image. At the same time, the optical image is transformed into a pseudo-ISAR image by the second generator and then the pseudo-ISAR image is transformed into an optical reconstructed image by the first generator.

[0063] Step S120: Extract the scattering points from the original ISAR image and the corresponding reconstructed ISAR image to obtain the first scattering point position matrix of the original ISAR image and the second scattering point position matrix of the reconstructed ISAR image.

[0064] Step S130: The equality of the first scattering point position matrix and the second scattering point position matrix of the ISAR reconstructed image is used as a constraint condition. Based on the form of the augmented Lagrange operator, this constraint condition is added as a penalty function term to the original loss function to reconstruct the final loss function.

[0065] Step S140: Train the generator network according to the final loss function to obtain the trained generator network;

[0066] Step S150: Obtain the optical target image to be converted, and use the second generator in the trained generation network to convert the optical target image to obtain the corresponding ISAR converted image.

[0067] In step S100, ISAR original images and optical images of targets of the same category are acquired, where the target category can be aerospace targets (taking aerospace targets as an example in this paper), or ships, etc.

[0068] In this embodiment, the generator network is trained unsupervised, meaning that the original ISAR image and the optical image input into the generator network do not need to correspond one-to-one.

[0069] In step S110, the generator network employs a Recurrent Generative Adversarial Network (CycleGAN), which includes discriminators for both the ISAR image domain and the optical image domain. During training, the processing flow of the original ISAR image and the optical image within the network is as follows: Figure 2 As shown, during training, the original ISAR image and the optical image are input from the first generator and the second generator, respectively, to obtain the corresponding pseudo-optical image and pseudo-ISAR image. Then, the ISAR reconstructed image and the optical reconstructed image are obtained through the second generator and the first generator, respectively.

[0070] like Figure 2As shown, generator G represents the mapping from the ISAR image domain to the optical image domain; generator F represents the mapping from the optical image domain to the ISAR image domain; D X D Y These represent the discriminator in the ISAR image domain and the discriminator in the optical image domain, respectively.

[0071] In one embodiment, generators G and F are implemented using CNNs based on upsampling and downsampling, and the two generators have the same network structure; the discriminator is implemented using PatchGAN.

[0072] Specifically, in CycleGAN, each network can be defined as follows:

[0073] For generators G and F:

[0074] G = CNN(x, θ) G (1)

[0075] F = CNN(y, θ) F (2)

[0076] For discriminant D X D Y This can be expressed as:

[0077]

[0078]

[0079] In formulas (1)-(4), x represents the input ISAR original image, y represents the input optical image, and θ represents the network parameters of the generator or discriminator.

[0080] In this embodiment, scattering points are extracted from both the original ISAR image and the reconstructed ISAR image, and added to the loss function. Based on the original cyclic consistency loss, the consistency constraint of the ISAR scattering point distribution model is introduced. This means that in addition to the original input-output pixel data consistency, the generator network also needs to satisfy the consistency of the input and output ISAR images in the feature space of scattering point distribution. This enables the generator network to learn how to learn the ISAR scattering point distribution.

[0081] In step S120, when extracting scattering points from the original ISAR image and the corresponding reconstructed ISAR image, the scattering points are extracted from the ISAR image using the Otsu's Inter-Class Variance (OSTU) method and a network method. Specifically, the grayscale threshold of the ISAR image is calculated using the Otsu's Inter-Class Variance method. This grayscale threshold allows for the segmentation of the ISAR image to achieve the highest foreground and background separation. Then, the scattering points of the ISAR image are extracted using the network method and the grayscale threshold.

[0082] Specifically, the calculation of the grayscale threshold of an ISAR image using the Otsu's method includes: For a grayscale image I, assuming the image has N pixels, the image grayscale level is L, and the number of pixels with grayscale level i is n. i Then the histogram distribution of the image is:

[0083]

[0084]

[0085] The image is divided into two categories based on grayscale levels using a threshold T: foreground (target) C0 and background C1. The proportion of foreground pixels in the entire image is ω0, with an average grayscale value of μ0; the proportion of background pixels is ω1, with an average grayscale value of μ1. The relevant formulas are expressed as follows:

[0086]

[0087]

[0088]

[0089]

[0090] Therefore, the average gray value μ of the image T It can be represented as:

[0091]

[0092] Define the inter-class variance of the sample points corresponding to the foreground and background in an image. for:

[0093]

[0094] The goal of the OTU method is to find a suitable threshold T that maximizes the foreground-background separation when segmenting the image based on T, i.e., maximizing the inter-class variance. Finally, the threshold T is found by iterating through the image's grayscale values.

[0095] Furthermore, the extraction of scattering points from the ISAR image based on the network method and grayscale thresholding includes: dividing the ISAR image into multiple networks of the same size, extracting pixels with amplitude values ​​greater than the grayscale threshold T from each network as scattering points, and recording the corresponding coordinates and amplitude values ​​of each scattering point. The scattering point position matrix is ​​then reconstructed based on the recorded coordinates of each scattering point in the original coordinate order.

[0096] Specifically, firstly, a 2×2 grid is divided on a 256×256 ISAR image. Then, using the amplitude value of the scattering point intensity in the ISAR image as a metric, a threshold is set using the OTU method. Based on this, points with amplitude values ​​greater than the threshold are extracted from all grid regions as scattering points. The coordinates and amplitude values ​​of these points are then recorded, and the extracted scattering point coordinates are reconstructed in the original coordinate order.

[0097] After extracting scattering points from ISAR images using the OTU method and the grid method, a 256×256 position matrix U is finally obtained. Each element in the position matrix U has a value in the interval [0,1], where points with non-zero values ​​represent extracted scattering points, such as... Figure 3 The diagram shows a scattering point location matrix extracted using the OTU method and the grid method. Figure 3 The results show that the OTU and grid method can extract scattering points from ISAR images and also extract the aircraft contours. Meanwhile, the original ISAR image itself contains cross-shaped sidelobe noise with relatively large amplitude. To ensure the realism of the converted image, the sidelobe noise is also extracted, which is beneficial for network training.

[0098] Let the input ISAR image be x. The Q(x) function represents the process of OTU calculating the image threshold. Simultaneously, the grid method for extracting scattering points can be understood as performing soft thresholding on the ISAR image, which is also equivalent to transforming the ISAR matrix. Therefore, the first scattering point position matrix U is represented as:

[0099] U = soft(x / Q(x), 0) (13)

[0100] In step S130, after extracting the scattering point location matrices from the original ISAR image and the reconstructed ISAR image, the extracted scattering point distribution is mathematically modeled and added as a constraint to the loss function during network training. In this embodiment, after the scattering point extraction, the first scattering point location matrix and the second scattering point location matrix of the input original ISAR image and the reconstructed ISAR image should be consistent, thereby constructing the constraint:

[0101] U=U′ (14)

[0102] In formula (14), U' represents the second scattering point position matrix obtained after scattering point extraction from the ISAR reconstructed image. The specific mathematical form of U' is as follows:

[0103] U'=soft(F(G(x)) / Q(F(G(x))),0) (15)

[0104] In the above formula, G(·) represents the transformation from the ISAR image domain to the optical image domain, and F(·) represents the transformation from the optical image domain to the ISAR image domain.

[0105] The original loss function of the generator network is:

[0106]

[0107] In formula (16), L GAN (G,D Y (x,y) represents the generator G and the discriminator D. Y The antagonistic loss, L GAN (F,D X (Y,X) represents the generator F and the discriminator D. X The adversarial loss, and λ cyc L cyc (G,F) represents the reconstruction loss function between the optical image and the optically reconstructed image, where x represents the input ISAR original image and y represents the input optical image. When training the generator network using the above original loss function, the training objective is to ensure that the first scattering point position matrix and the second scattering point position matrix are consistent; that is, training stops when U = U'.

[0108] In this embodiment, to ensure the convexity of the problem and reduce the difficulty of solving it, the constraints are added to the loss function in the form of an augmented Lagrange operator. That is, the constraints are added to the optimization problem as a penalty function term, resulting in the final loss function:

[0109]

[0110] In formula (17), λ scatter Represents the Lagrange multiplier, ρ scatter This represents the coefficient of the penalty function term. The improved loss function is still differentiable, where U' in the augmented Lagrangian function is obtained through generators G and F, which is the second scattering point position matrix, containing the network parameters of the two generators. Therefore, the network parameters can be updated through gradient backpropagation.

[0111] Therefore, training the generator network using the final loss function to obtain a trained generator network also includes: calculating the gradient of the final loss function, and adjusting the parameters of the first and second generators in the generator network according to the calculation results, until convergence, thus obtaining the trained generator network. Figure 4 The diagram shown illustrates the convergence of the loss function during the training of the generative network. Figure 4 It can be clearly seen that after adding the ISAR scattering point distribution constraint, the model converges faster and with higher accuracy, verifying that the added physical property constraint effectively constrains the solution space of the model parameter optimization.

[0112] Specifically, the parameters of the generator network are updated based on the Adam optimization algorithm. This ensures that the physical characteristic model of the scattering points in the ISAR reconstructed image generated by the generator network remains consistent with that of the original ISAR image after scattering point extraction. This embeds prior knowledge of ISAR physical characteristics into the optimization of network parameters, guaranteeing that the ISAR image generated from the optical input possesses the scattering point distribution characteristics.

[0113] After iterative training of the generator network, the resulting second generator has the ability to convert optical images into ISAR images with physical characteristics.

[0114] It should be noted that in this embodiment, supervised training is used when training the generator network, meaning that the original ISAR image and the optical image input into the generator network are not in one-to-one correspondence.

[0115] In step S150, when the trained generator network is used to convert the optical image to be converted, the optical image to be converted is input into the second generator, and the output is the ISAR converted image corresponding to the optical image to be converted. For example... Figure 5 The image shown is a schematic diagram of the optical-ISAR image conversion result obtained using this method. Figure 5 As can be seen, after incorporating the physical information extracted from the scattering points, the converted pseudo-ISAR image possesses a certain degree of realism and can generate cross-shaped sidelobe noise similar to the original ISRA image. This demonstrates that introducing the scattering point model has achieved certain results, especially in the conversion between optical and ISAR images. The model no longer simply generates scattering points uniformly according to the contour of the optical image, but rather learns to plan the distribution of scattering points based on the target's attitude and orientation, enabling the generated ISAR image to meet the requirements of realism in terms of the physical characteristics of the scattering points.

[0116] In this embodiment, optical-ISAR image conversion in multiple directions using this method is also provided, such as... Figure 6As shown, the results demonstrate that the conversion from optical images to ISAR images has achieved initial success. However, since the change in the aircraft target orientation in the optical image cannot represent the actual changes in the aircraft target's pitch and azimuth angles, the resulting pseudo-ISAR images still cannot fully reflect the real radar ISAR imaging results, lacking the ability to learn about the intensity characteristics of the scattering center, the distribution characteristics at different attitude angles, and the different learning processes.

[0117] In this embodiment, the conversion of unfamiliar satellite targets from optical-ISAR images using this method is also provided, such as... Figure 7 As shown in the figure, the generative model trained on the satellite target dataset generates ISAR data on simulated optical images of unfamiliar satellites. It can be seen that the model, based on the paired training dataset, performs well in data generation for similar targets, fully representing the scattering point distribution of the satellite target, especially the dense scattering point distribution characteristics on the entire solar panel and its main components.

[0118] In the aforementioned optical-radar ISAR image conversion method based on generative networks, the extracted ISAR image scattering points are introduced into the training of a recurrent generative adversarial network, enabling it to convert optical images into ISAR images with physical characteristics. This method can provide ISAR image data with a certain degree of credibility, providing data support for it. Furthermore, the generative network has a certain generalization performance and can convert unfamiliar optical targets.

[0119] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but may be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0120] In one embodiment, such as Figure 8 As shown, an optical-radar ISAR image conversion device based on a generative network is provided, including: a training set acquisition module 200, an image domain conversion module 210, a scattering point location matrix reconstruction module 220, a loss function reconstruction module 230, a network training module 240, and an optical-radar ISAR image conversion module 250, wherein:

[0121] The training set acquisition module 200 is used to acquire an image training set, which includes multiple ISAR original images and optical images with the same target category.

[0122] Image domain conversion module 210 is used to input the original ISAR image and optical image from the image training set into the generation network. The original ISAR image is mapped from the ISAR image domain to the optical image domain by the first generator in the generation network and transformed into a corresponding pseudo-optical image. Then, the pseudo-optical image is mapped from the optical image domain to the ISAR image domain by the second generation device and transformed into a corresponding ISAR reconstructed image. At the same time, the optical image is transformed into a pseudo-ISAR image by the second generator and then the pseudo-ISAR image is transformed into an optical reconstructed image by the first generation device.

[0123] The scattering point position matrix reconstruction module 220 is used to extract the scattering points of the original ISAR image and the corresponding reconstructed ISAR image respectively, to obtain the first scattering point position matrix of the original ISAR image and the second scattering point position matrix of the reconstructed ISAR image.

[0124] The loss function reconstruction module 230 is used to take the equality of the first scattering point position matrix and the second scattering point position matrix of the ISAR reconstructed image as a constraint condition, and add the constraint condition as a penalty function term to the original loss function based on the form of the augmented Lagrange operator to reconstruct the final loss function.

[0125] The network training module 240 is used to train the generator network according to the final loss function to obtain the trained generator network.

[0126] The optical-radar ISAR image conversion module 250 is used to acquire an optical target image to be converted, and to convert the optical target image to be converted using the second generator in the trained generation network to obtain the corresponding ISAR converted image.

[0127] Specific limitations regarding the generative network-based optical-radar ISAR image conversion device can be found in the limitations of the generative network-based optical-radar ISAR image conversion method described above, and will not be repeated here. Each module in the aforementioned generative network-based optical-radar ISAR image conversion device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in a computer device, or stored in software in the memory of a computer device, so that the processor can call and execute the corresponding operations of each module.

[0128] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements an optical-radar ISAR image conversion method based on generative networks. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0129] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0130] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0131] Obtain an image training set, which includes multiple ISAR original images and optical images with the same target category;

[0132] The original ISAR image and the optical image in the image training set are input into the generation network. The original ISAR image is mapped from the ISAR image domain to the optical image domain by the first generator in the generation network and transformed into a corresponding pseudo-optical image. Then, the pseudo-optical image is mapped from the optical image domain to the ISAR image domain by the second generator and transformed into a corresponding ISAR reconstructed image. At the same time, the optical image is transformed into a pseudo-ISAR image by the second generator and then the pseudo-ISAR image is transformed into an optical reconstructed image by the first generator.

[0133] Scattering points are extracted from the original ISAR image and the corresponding reconstructed ISAR image to obtain the first scattering point position matrix of the original ISAR image and the second scattering point position matrix of the reconstructed ISAR image.

[0134] The first scattering point position matrix and the second scattering point position matrix of the ISAR reconstructed image are made equal as a constraint condition. Based on the augmented Lagrange operator, the constraint condition is added as a penalty function term to the original loss function to reconstruct the final loss function.

[0135] The generator network is trained based on the final loss function to obtain a trained generator network;

[0136] An optical target image to be converted is acquired, and the second generator in the trained generation network is used to convert the optical target image to obtain the corresponding ISAR converted image.

[0137] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0138] Obtain an image training set, which includes multiple ISAR original images and optical images with the same target category;

[0139] The original ISAR image and the optical image in the image training set are input into the generation network. The original ISAR image is mapped from the ISAR image domain to the optical image domain by the first generator in the generation network and transformed into a corresponding pseudo-optical image. Then, the pseudo-optical image is mapped from the optical image domain to the ISAR image domain by the second generator and transformed into a corresponding ISAR reconstructed image. At the same time, the optical image is transformed into a pseudo-ISAR image by the second generator and then the pseudo-ISAR image is transformed into an optical reconstructed image by the first generator.

[0140] Scattering points are extracted from the original ISAR image and the corresponding reconstructed ISAR image to obtain the first scattering point position matrix of the original ISAR image and the second scattering point position matrix of the reconstructed ISAR image.

[0141] The first scattering point position matrix and the second scattering point position matrix of the ISAR reconstructed image are made equal as a constraint condition. Based on the augmented Lagrange operator, the constraint condition is added as a penalty function term to the original loss function to reconstruct the final loss function.

[0142] The generator network is trained based on the final loss function to obtain a trained generator network;

[0143] An optical target image to be converted is acquired, and the second generator in the trained generation network is used to convert the optical target image to obtain the corresponding ISAR converted image.

[0144] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0145] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0146] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A generative network-based optical-radar ISAR image conversion method, characterized in that, The method includes: Obtain an image training set, which includes multiple ISAR original images and optical images with the same target category; The original ISAR image and the optical image in the image training set are input into the generation network. The original ISAR image is mapped from the ISAR image domain to the optical image domain by the first generator in the generation network and transformed into a corresponding pseudo-optical image. Then, the pseudo-optical image is mapped from the optical image domain to the ISAR image domain by the second generator and transformed into a corresponding ISAR reconstructed image. At the same time, the optical image is transformed into a pseudo-ISAR image by the second generator and then the pseudo-ISAR image is transformed into an optical reconstructed image by the first generator. Scattering points are extracted from the original ISAR image and the corresponding reconstructed ISAR image to obtain the first scattering point position matrix of the original ISAR image and the second scattering point position matrix of the reconstructed ISAR image. The first scattering point position matrix and the second scattering point position matrix of the ISAR reconstructed image are made equal as a constraint condition. Based on the augmented Lagrange operator, the constraint condition is added as a penalty function term to the original loss function to reconstruct the final loss function. The generator network is trained based on the final loss function to obtain a trained generator network; An optical target image to be converted is acquired, and the second generator in the trained generation network is used to convert the optical target image to obtain the corresponding ISAR converted image.

2. The optical-radar ISAR image conversion method according to claim 1, characterized in that, When extracting scattering points from the original ISAR image and the corresponding reconstructed ISAR image, the scattering points of the ISAR image are extracted using the maximum inter-class variance method and the network method.

3. The optical-radar ISAR image conversion method according to claim 2, characterized in that, The extraction of scattering points from ISAR images based on the Otsu's method and the network method includes: The grayscale threshold of the ISAR image is calculated based on the maximum inter-class variance method. The foreground and background separation is obtained by segmenting the ISAR image based on the grayscale threshold. Then, the scattering points of the ISAR image are extracted using the network method and grayscale threshold.

4. The optical-radar ISAR image conversion method according to claim 3, characterized in that, The extraction of scattering points from ISAR images based on network methods and grayscale thresholds includes: The ISAR image is divided into multiple networks of the same size; Pixels with amplitude values ​​greater than the grayscale threshold are extracted from each network as scattering points, and the corresponding coordinates and amplitude values ​​of each scattering point are recorded. The scattering point position matrix is ​​obtained by reconstructing the corresponding coordinates of each scattering point according to the original coordinate order.

5. The optical-radar ISAR image conversion method according to claim 4, characterized in that, The generating network employs a recurrent generative adversarial network and also includes a discriminator in the ISAR image domain and a discriminator in the optical image domain. The first generator and the second generator are implemented using a CNN structure, and the two discriminators are implemented using a PatchGAN structure.

6. The optical-radar ISAR image conversion method according to claim 5, characterized in that, The original loss function is: In the above equation, generator G represents the mapping from the ISAR image domain to the optical image domain, and generator F represents the mapping from the optical image domain to the ISAR image domain. , These represent the discriminator in the ISAR image domain and the discriminator in the optical image domain, respectively. Represents generator and discriminator The antagonistic losses, Represents generator and discriminator The antagonistic losses, and This represents the reconstruction loss function between the optical image and the optically reconstructed image. This represents the original ISAR image as input. This represents the input optical image.

7. The optical-radar ISAR image conversion method according to claim 6, characterized in that, The final loss function is: In the above formula, Represents the Lagrange multipliers. This represents the coefficient of the penalty function term.

8. The optical-radar ISAR image conversion method according to claim 7, characterized in that, Training the generator network based on the final loss function to obtain the trained generator network further includes: The gradient of the final loss function is calculated, and the parameters of the first generator and the second generator in the generator network are corrected according to the direction of the calculation result until convergence, thus obtaining the trained generator network.

9. An optical-radar ISAR image conversion device based on a generative network, characterized in that, The device includes: The training set acquisition module is used to acquire an image training set, which includes multiple ISAR original images and optical images with the same target category. The image domain conversion module is used to input the original ISAR image and the optical image from the image training set into the generation network. The original ISAR image is mapped from the ISAR image domain to the optical image domain by the first generator in the generation network and transformed into a corresponding pseudo-optical image. Then, the pseudo-optical image is mapped from the optical image domain to the ISAR image domain by the second generation device and transformed into a corresponding ISAR reconstructed image. At the same time, the optical image is transformed into a pseudo-ISAR image by the second generator and then the pseudo-ISAR image is transformed into an optical reconstructed image by the first generation device. The scattering point position matrix reconstruction module is used to extract the scattering points of the original ISAR image and the corresponding reconstructed ISAR image, respectively, to obtain the first scattering point position matrix of the original ISAR image and the second scattering point position matrix of the reconstructed ISAR image. The loss function reconstruction module is used to take the equality of the first scattering point position matrix and the second scattering point position matrix of the ISAR reconstructed image as a constraint condition, and add the constraint condition as a penalty function term to the original loss function based on the form of augmented Lagrange operator to reconstruct the final loss function. The network training module is used to train the generator network according to the final loss function to obtain the trained generator network. The optical-radar ISAR image conversion module is used to acquire the optical target image to be converted, and to convert the optical target image to be converted using the second generator in the trained generation network to obtain the corresponding ISAR converted image.

Images