Data physical dual-drive SAR (Synthetic Aperture Radar) image generation method and device and electronic equipment

Abstract

The invention relates to a data physical dual-drive SAR image generation method and device and electronic equipment, and the method comprises the steps: obtaining an SAR image sequence and a visible light image sequence which meet preset conditions; performing three-dimensional reconstruction on the visible light image sequence to obtain an initial three-dimensional grid, and endowing the initial three-dimensional grid with corresponding material parameters to obtain an actual three-dimensional grid with material information; and performing process simulation of SAR finite physical modeling on the actual three-dimensional grid according to the parameter corresponding to each image in the image sequence to obtain a physical simulation image sequence, and generating a corresponding SAR image based on the physical simulation image sequence and the image sequence. Therefore, the technical problem that the development of deep learning in SAR image application is limited due to the fact that the generalization ability of an existing SAR image generation model driven by a physical or simulated scene is poor when facing an actual scene in the related technology is solved.

Description

Technical Field

[0001] The present application relates to the field of remote sensing imaging technology, and in particular to a data-physics dual-driven SAR image generation method, device and electronic equipment. Background Art

[0002] With the development of remote sensing technology, SAR (Synthetic Aperture Radar) has gradually become an important tool for earth observation and environmental monitoring, playing an important role in military reconnaissance, disaster monitoring and urban planning. In these applications, ATR (Automatic Target Recognition) technology can quickly and accurately identify and classify targets in complex environments, providing timely support for decision-making. The effective application of deep learning in SAR ATR depends on high-quality data sets. The high cost of acquiring SAR data and the cumbersome image annotation process make it challenging to build a large-scale, high-quality training data set. Therefore, generating high-quality SAR images becomes the key to solving this problem.

[0003] In the related technologies, one type of SAR simulation method is designed according to specific application scenarios, such as GRECOSAR (Graphical Electromagnetic Computing SAR) which generates polarized SAR images by solving electromagnetic problems through high-frequency approximation, cSAR which uses scattering models to simulate multi-channel signals of SAR platforms and moving targets under arbitrary trajectories in the time domain, point target reference spectrum which is used to calculate bistatic SAR data in the frequency domain, and time domain and frequency domain hybrid methods which can simulate the original data of moving targets with speckle noise. Another type of technology is based on the imaging principle of SAR for simulation, such as RaySAR which uses ray tracing technology to simulate the images generated by echo signals with different bounce times, and combining geometric optics methods with electromagnetic simulation can balance simulation speed and image accuracy according to actual needs. Traditional SAR image simulation methods mainly rely on the understanding of the physical process of SAR imaging. However, in actual situations, the imaging process involves a variety of complex interactions. Except for the electromagnetic simulation method, there is a large gap between the results obtained by other traditional methods and the actual data. The electromagnetic simulation method has great difficulties in practical applications due to its long calculation time.

[0004] Deep learning technology has shown great potential in the field of SAR image generation. Currently, most of the related technologies focus on using generative adversarial networks (GANs) to achieve data expansion. However, due to the limitations of the number and quality of public datasets in the SAR field, most of the related technologies are based on the MSTAR dataset, which has a simple background and mainly collects SAR images of vehicle targets. This results in poor generalization of the model when facing complex scenes such as cities. This limitation restricts the development of deep learning in a wider range of SAR image applications and needs to be improved urgently. Summary of the invention

[0005] The present application provides a data-physics dual-driven SAR image generation method, device and electronic device to solve the technical problem in the related art that the existing physical or simulated scene-driven SAR image generation model has poor generalization ability when facing actual scenes, thereby limiting the development of deep learning in SAR image applications.

[0006] A first aspect of the present application provides a data-physics dual-driven SAR image generation method, comprising the following steps: acquiring a SAR image sequence and a visible light image sequence that meet preset conditions; performing three-dimensional reconstruction on the visible light image sequence to obtain an initial three-dimensional grid, and assigning corresponding material parameters to the initial three-dimensional grid to obtain an actual three-dimensional grid with material information; performing process simulation of SAR finite physical modeling on the actual three-dimensional grid according to parameters corresponding to each image in the image sequence to obtain a physically simulated image sequence, and generating corresponding SAR images based on the physically simulated image sequence and the image sequence.

[0007] Optionally, in one embodiment of the present application, assigning corresponding material parameters to the initial three-dimensional mesh to obtain an actual three-dimensional mesh with material information includes: performing panoramic segmentation on the initial three-dimensional mesh to obtain corresponding segmentation results; and using the segmentation results to assign corresponding material parameters to the initial three-dimensional mesh to obtain the actual three-dimensional mesh.

[0008] Optionally, in one embodiment of the present application, the performing panoramic segmentation on the initial three-dimensional mesh to obtain a corresponding segmentation result includes: performing point clouding on the initial three-dimensional mesh to obtain a corresponding point cloud set; and performing panoramic segmentation on the point cloud set using a pre-trained point cloud segmentation model to obtain the segmentation result.

[0009] Optionally, in one embodiment of the present application, generating a corresponding SAR image based on the physical simulation image sequence and the image sequence includes: constructing a simulation-actual image pair data set using the physical simulation image sequence and the image sequence; performing data processing on the simulation-actual image pair data set to obtain a processed image; dividing the processed image into multiple image blocks using a preset division principle; using the multiple image blocks as network input to generate corresponding actual image blocks using a pre-trained neural network; and splicing the actual image blocks according to the original image positions to obtain the SAR image.

[0010] Optionally, in one embodiment of the present application, the data processing of the simulation-actual image pair data set to obtain a processed image includes: overall standardization of the simulation-actual image pair data set and elimination of outliers in the simulation-actual image pair data set to obtain a preliminary processed image; and normalization of the preliminary processed image to obtain the processed image.

[0011] Optionally, in one embodiment of the present application, before using a pre-trained neural network to generate a corresponding actual image block, it also includes: using a sample simulated SAR image data set and a corresponding sample SAR image data set to train the neural network, and calculating a corresponding loss function; optimizing the neural network based on the loss function until the neural network meets a preset convergence condition, and obtaining the pre-trained neural network.

[0012] A second aspect of the present application provides a data-physics dual-driven SAR image generation device, comprising: an acquisition module, used to acquire a SAR image sequence and a visible light image sequence that meet preset conditions; a reconstruction module, used to perform three-dimensional reconstruction on the visible light image sequence to obtain an initial three-dimensional grid, and assign corresponding material parameters to the initial three-dimensional grid to obtain an actual three-dimensional grid with material information; a generation module, used to perform a process simulation of SAR finite physical modeling on the actual three-dimensional grid according to the parameters corresponding to each image in the image sequence to obtain a physical simulation image sequence, and generate corresponding SAR images based on the physical simulation image sequence and the image sequence.

[0013] Optionally, in one embodiment of the present application, the reconstruction module includes: a segmentation unit, used to perform panoramic segmentation on the initial three-dimensional mesh to obtain a corresponding segmentation result; and an assignment unit, used to assign corresponding material parameters to the initial three-dimensional mesh using the segmentation result to obtain the actual three-dimensional mesh.

[0014] Optionally, in one embodiment of the present application, the segmentation unit includes: a first processing subunit, used to perform point clouding on the initial three-dimensional grid to obtain a corresponding point cloud set; and a segmentation subunit, used to perform panoramic segmentation on the point cloud set using a pre-trained point cloud segmentation model to obtain the segmentation result.

[0015] Optionally, in one embodiment of the present application, the generation module includes: a construction unit, used to construct a simulation-actual image pair data set using the physical simulation image sequence and the image sequence; a first processing unit, used to perform data processing on the simulation-actual image pair data set to obtain a processed image; a division unit, used to divide the processed image into multiple image blocks using a preset division principle; a generation unit, used to use the multiple image blocks as network input to generate corresponding actual image blocks using a pre-trained neural network; and a splicing unit, used to splice the actual image blocks according to the original image positions to obtain the SAR image.

[0016] Optionally, in one embodiment of the present application, the processing unit includes: a second processing subunit, used to perform overall standardization on the simulation-actual image pair data set and eliminate outliers in the simulation-actual image pair data set to obtain a preliminary processed image; and a third processing subunit, used to normalize the preliminary processed image to obtain the processed image.

[0017] Optionally, in one embodiment of the present application, the generation module also includes: a training unit, used to train the neural network using a sample simulated SAR image data set and a corresponding sample SAR image data set, and calculate a corresponding loss function; an optimization unit, used to optimize the neural network based on the loss function until the neural network meets a preset convergence condition and obtains the pre-trained neural network.

[0018] A third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the data-physics dual-driven SAR image generation method as described in the above embodiment.

[0019] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the data-physics dual-driven SAR image generation method as described in the above embodiments.

[0020] A fifth aspect of the present application provides a computer program product, including a computer program, which, when executed, is used to implement the above data-physics dual-driven SAR image generation method.

[0021] The embodiment of the present application can directly obtain the target three-dimensional model through the collected visible light information and automatically assign the corresponding material information through the panoramic segmentation method, which greatly reduces the cost of manual modeling and material annotation. In terms of physics, a method with high computational efficiency is used to maintain the accuracy of the main structure of the generated image. In terms of data, a neural network is used to learn the difference between physical simulation images and real images, emphasizing fine structures and image noise to obtain more realistic SAR images. In this way, the technical problem that the existing physical or simulated scene-driven SAR image generation model in the related art has poor generalization ability when facing actual scenes, thereby limiting the development of deep learning in SAR image applications is solved.

[0022] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0024] Figure 1 A flowchart of a data-physics dual-driven SAR image generation method provided according to an embodiment of the present application;

[0025] Figure 2 A schematic diagram of a Unet network structure according to an embodiment of the present application;

[0026] Figure 3 It is a principle schematic diagram of a data-physics dual-driven SAR image generation method according to an embodiment of the present application;

[0027] Figure 4 A schematic diagram of the NeRF2Mesh structure according to an embodiment of the present application;

[0028] Figure 5 A schematic diagram of the SAM2POINT model structure according to an embodiment of the present application;

[0029] Figure 6 A schematic diagram of the structure of a data-physics dual-driven SAR image generation device provided according to an embodiment of the present application;

[0030] Figure 7 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0032] The following describes the data-physics dual-driven SAR image generation method, device and electronic device of the embodiment of the present application with reference to the accompanying drawings. In view of the technical problem that the existing physical or simulated scene-driven SAR image generation model in the related art mentioned in the above background technology has poor generalization ability when facing actual scenes, thereby limiting the development of deep learning in SAR image applications, the present application provides a data-physics dual-driven SAR image generation method, in which the target three-dimensional model can be directly obtained through the collected visible light information and the corresponding material information can be automatically assigned through the panoramic segmentation method, which greatly reduces the cost of manual modeling and material annotation. In terms of physics, a method with high computational efficiency is used to maintain the accuracy of the main structure of the generated image. In terms of data, a neural network is used to learn the difference between physical simulation images and real images, emphasizing fine structures and image noise to obtain more realistic SAR images. As a result, the technical problem that the existing physical or simulated scene-driven SAR image generation model in the related art has poor generalization ability when facing actual scenes, thereby limiting the development of deep learning in SAR image applications is solved.

[0033] Specifically, Figure 1 A schematic flow chart of a data-physics dual-driven SAR image generation method provided in an embodiment of the present application.

[0034] like Figure 1 As shown, the data-physics dual-driven SAR image generation method includes the following steps:

[0035] In step S101, a SAR image sequence and a visible light image sequence that meet preset conditions are acquired.

[0036] In the actual implementation process, the embodiment of the present application can obtain a SAR image sequence of the same direction at the same time through data acquisition. and visible light image sequences Where T represents the total number of images contained in the sequence, and H×W represents the size of the image.

[0037] In step S102, three-dimensional reconstruction is performed on the visible light image sequence to obtain an initial three-dimensional mesh, and corresponding material parameters are assigned to the initial three-dimensional mesh to obtain an actual three-dimensional mesh with material information.

[0038] As a possible implementation method, the embodiment of the present application can perform three-dimensional reconstruction on the visible light image sequence to obtain an initial three-dimensional grid:

[0039] M=R(I opt )

[0040] Among them, R is the reconstruction function, and M is the three-dimensional model based on grid representation obtained after reconstruction, that is, the initial three-dimensional grid.

[0041] Furthermore, the embodiment of the present application can assign corresponding material parameters to the initial three-dimensional mesh, thereby obtaining an actual three-dimensional mesh with material information. The embodiment of the present application directly reconstructs the target three-dimensional model through the collected visible light and automatically assigns corresponding material information through the panoramic segmentation method, which greatly reduces the cost of manual modeling and material annotation.

[0042] Optionally, in one embodiment of the present application, corresponding material parameters are assigned to the initial three-dimensional mesh to obtain an actual three-dimensional mesh with material information, including: performing panoramic segmentation on the initial three-dimensional mesh to obtain corresponding segmentation results; and using the segmentation results to assign corresponding material parameters to the initial three-dimensional mesh to obtain an actual three-dimensional mesh.

[0043] In some embodiments, the initial three-dimensional mesh may be panoptically segmented, and the segmented product may be used to assign corresponding material parameters to the initial three-dimensional mesh, thereby automatically assigning material information and reducing the cost of manual modeling and material annotation.

[0044] Optionally, in one embodiment of the present application, panoramic segmentation is performed on the initial three-dimensional mesh to obtain a corresponding segmentation result, including: point clouding the initial three-dimensional mesh to obtain a corresponding point cloud set; and performing panoramic segmentation on the point cloud set using a pre-trained point cloud segmentation model to obtain a segmentation result.

[0045] For example, in an embodiment of the present application, the panoptic segmentation of the initial three-dimensional network may include:

[0046] In the embodiment of the present application, the initial three-dimensional grid M can be converted into a point cloud:

[0047] P in =f(M)

[0048] Among them, P in is the point cloud set obtained after point cloudification, N point Represents the total number of points in the point cloud, and f is the point cloud function. Use the pre-trained point cloud segmentation model φ p P in Perform panoptic segmentation:

[0049] P out =φ p (Pin )

[0050] Among them, P out = {P0,P1,...,P K} is the point cloud set after segmentation, P i represents the point cloud set of the i-th semantic category, and K represents the total number of point cloud semantic categories.

[0051] The present application embodiment can be based on P out Assign the corresponding material parameters to the mesh body M to obtain a three-dimensional mesh M with material information * , that is, the actual three-dimensional grid.

[0052] In step S103, the process of finite physical modeling of SAR is simulated for the actual three-dimensional grid according to the parameters corresponding to each image in the image sequence to obtain a physical simulation image sequence, and a corresponding SAR image is generated based on the physical simulation image sequence and the image sequence.

[0053] The embodiments of the present application can be based on a physics-driven simulation platform, such as RaySAR, OpenSARSim, etc. SAR The parameter pair M corresponding to each image in * Perform SAR finite physical modeling process simulation to obtain physical simulation image sequence I sim , and generate corresponding SAR images based on the physical simulation image sequence and image sequence.

[0054] In the physics aspect, the embodiments of the present application can use a method with high computational efficiency to maintain the main structural accuracy of the generated image, and in the data aspect, use a neural network to learn the difference between the physical simulation image and the real image, emphasizing the fine structure and image noise to obtain a more realistic SAR image.

[0055] Optionally, in one embodiment of the present application, a corresponding SAR image is generated based on a physical simulation image sequence and an image sequence, including: constructing a simulation-actual image pair data set using the physical simulation image sequence and the image sequence; performing data processing on the simulation-actual image pair data set to obtain a processed image; dividing the processed image into multiple image blocks using a preset division principle; using the multiple image blocks as network input to generate corresponding actual image blocks using a pre-trained neural network; and splicing the actual image blocks according to the positions of the original images to obtain a SAR image.

[0056] The embodiment of the present application can construct a simulation-actual image pair (I sim ,I SAR ) dataset, and the simulated-real image pairs (I sim ,I SAR) data set to obtain the processed image, and divide the processed image into N image blocks patch Blocks, wherein the division principle is that the total field of view of the image blocks divided from the same image can cover the entire simulated SAR image.

[0057] In this embodiment of the present application, the image block can be As network input, a neural network is used to generate the generated image block p out :

[0058] p out =φ unet (p g )

[0059] After the generation is completed, the image blocks are spliced ​​according to the original image position, and the embodiment of the present application can generate a SAR image.

[0060]

[0061] Optionally, in one embodiment of the present application, data processing is performed on the simulation-actual image pair data set to obtain a processed image, including: overall standardization of the simulation-actual image pair data set and elimination of outliers in the simulation-actual image pair data set to obtain a preliminary processed image; normalization of the preliminary processed image to obtain a processed image.

[0062] In some embodiments, the simulated-actual image pair (I sim ,I SAR ) The data set is standardized as a whole and outliers are removed, and then normalized to obtain the processed image.

[0063] Optionally, in one embodiment of the present application, before using a pre-trained neural network to generate a corresponding actual image block, it also includes: using a sample simulated SAR image data set and a corresponding sample SAR image data set to train the neural network, and calculating a corresponding loss function; optimizing the neural network based on the loss function until the neural network meets a preset convergence condition, and obtaining a pre-trained neural network.

[0064] For example, the present application embodiment may use a neural network φ of a UNet structure unct For training, the structure is as follows Figure 2 As shown, the loss function f l for:

[0065]

[0066] Where y is the target image, is the image output by the model, and n is the number of samples.

[0067] Combination Figures 2 to 5 As shown, the working principle of the data-physics dual-driven SAR image generation method of the embodiment of the present application is described in detail with an embodiment.

[0068] like Figure 3 As shown, it is a schematic diagram of the principle of an embodiment of the present application. In terms of physics, the embodiment of the present application can use a method with high computational efficiency to maintain the main structural accuracy of the generated image. In terms of data, a neural network is used to learn the difference between a physical simulation image and a real image, emphasizing fine structure and image noise to obtain a more realistic SAR image.

[0069] In the actual implementation process, for N static scanning environments, the embodiment of the present application can use the circular scanning mode to scan one circle each, use the drone platform equipped with multi-source information acquisition to collect signals at 200 angles and process them to obtain a SAR image sequence with an image size of H×W. and visible light image sequences

[0070] In the embodiment of the present application, NeRF2Mesh can be used to perform three-dimensional reconstruction of a visible light image sequence.

[0071] Among them, NeRF2Mesh is a 3D reconstruction method based on NeRF structure. It reconstructs the mesh of the corresponding scene by inputting images of multiple perspectives of a scene. The structure can be as follows: Figure 4 shown.

[0072] The embodiments of the present application can be used for opt Use NeRF2Mesh to perform 3D reconstruction to obtain the scene mesh M seene , that is, the initial three-dimensional grid M scene .

[0073] In the embodiment of the present application, OpenSARSIM can be used for physical simulation.

[0074] Among them, OpenSARSIM is a simulation simulator for SAR images, which mainly uses PO (Physics Optics) and IAD (Incremental Length Diffraction Coefficients) methods to simulate high-frequency electromagnetic scattering. PO can calculate the scattering approximate component σ of the target surface element. PO , ILDC finds the diffraction approximate component σ of the target edge element ILDC , then the target backscatter coefficient σ is:

[0075] σ=σ PO +σ iLDC

[0076] When rendering the corresponding viewing angle image, the embodiment of the present application can project the scattering center onto the azimuth-range plane, and perform convolution calculation with the SINC function to obtain a scattering intensity distribution map.

[0077] For scene grid M scene After point clouding, the present application embodiment can use the SAM2POINT model to perform point cloud segmentation. The model can be as follows: Figure 5 As shown. In the embodiment of the present application, the corresponding material coefficients of the surface can be assigned according to the panoptic segmentation mask, including relative dielectric constant and relative magnetic permeability. Input the simulation view angle, radar parameters, scene grid M corresponding to the OpenSARSIM platform scene And material parameters to get the scattering intensity distribution map

[0078] The embodiment of the present application can construct a simulation-actual image pair data set (I g ,I SAR ), the image is cut into 512×512 blocks, and the neural network φ of the UNet structure is used unet For training, the structure is as follows Figure 2 As shown, the loss function f l for:

[0079]

[0080] Where y is the target image, is the image output by the model, and n is the number of samples.

[0081] When using the trained network, the embodiment of the present application can divide the simulated SAR image into N blocks of 512×512, and the principle of division is that the image blocks cover the entire simulated SAR image. As network input, a neural network is used to generate the generated image block p out :

[0082] p out =φ unet (p g )

[0083] After the generation is completed, the image blocks are spliced ​​according to the original image position, and the overlapping parts are averaged to obtain the generated SAR image.

[0084] In summary, the embodiments of the present application can directly reconstruct the target three-dimensional model through the collected visible light and automatically assign the corresponding material information through the panoramic segmentation method, which greatly reduces the cost of manual modeling and material labeling; the embodiments of the present application can be directly used in actual complex scenes, and the constructed data set is directly related to the actual data set. The trained generation model is more robust and the generated image is more realistic; the embodiments of the present application use the output image of the physical simulation as the input of the deep learning generation model, which has better physical consistency than the generation method that relies solely on data-driven.

[0085] According to the data-physics dual-driven SAR image generation method proposed in the embodiment of the present application, the target three-dimensional model can be directly obtained through the collected visible light information and the corresponding material information can be automatically assigned through the panoramic segmentation method, which greatly reduces the cost of manual modeling and material annotation. In terms of physics, a method with high computational efficiency is used to maintain the accuracy of the main structure of the generated image. In terms of data, a neural network is used to learn the difference between physical simulation images and real images, emphasizing fine structures and image noise to obtain more realistic SAR images. As a result, the technical problem in the related art that the existing physical or simulated scene-driven SAR image generation model has poor generalization ability when facing actual scenes, thereby limiting the development of deep learning in SAR image applications is solved.

[0086] Next, the data-physics dual-driven SAR image generation device proposed in the embodiment of the present application is described with reference to the accompanying drawings.

[0087] Figure 6 It is a block diagram of a data-physics dual-driven SAR image generation device according to an embodiment of the present application.

[0088] like Figure 6 As shown, the data-physics dual-driven SAR image generation device 10 includes: an acquisition module 100 , a reconstruction module 200 and a generation module 300 .

[0089] Specifically, the acquisition module 100 is used to acquire a SAR image sequence and a visible light image sequence that meet preset conditions.

[0090] The reconstruction module 200 is used to perform three-dimensional reconstruction on the visible light image sequence to obtain an initial three-dimensional mesh, and to assign corresponding material parameters to the initial three-dimensional mesh to obtain an actual three-dimensional mesh with material information.

[0091] The generation module 300 is used to simulate the process of finite physical modeling of SAR on the actual three-dimensional grid according to the parameters corresponding to each image in the image sequence to obtain a physical simulation image sequence, and generate corresponding SAR images based on the physical simulation image sequence and the image sequence.

[0092] Optionally, in one embodiment of the present application, the reconstruction module 200 includes: a segmentation unit and an assignment unit.

[0093] The segmentation unit is used to perform panorama segmentation on the initial three-dimensional grid to obtain corresponding segmentation results;

[0094] The assigning unit is used to assign corresponding material parameters to the initial three-dimensional mesh using the segmentation result to obtain the actual three-dimensional mesh.

[0095] Optionally, in one embodiment of the present application, the segmentation unit includes: a first processing subunit and a segmentation subunit.

[0096] The first processing subunit is used to convert the initial three-dimensional grid into a point cloud to obtain a corresponding point cloud set.

[0097] The segmentation subunit is used to perform panoramic segmentation on the point cloud set using a pre-trained point cloud segmentation model to obtain a segmentation result.

[0098] Optionally, in one embodiment of the present application, the generation module 300 includes: a construction unit, a first processing unit, a division unit, a generation unit and a splicing unit.

[0099] The construction unit is used to construct a simulation-actual image pair data set using a physical simulation image sequence and an image sequence.

[0100] The first processing unit is used to perform data processing on the simulation-actual image pair data set to obtain a processed image.

[0101] The dividing unit is used to divide the processed image into multiple image blocks using a preset dividing principle.

[0102] A generation unit is used to take a plurality of image patches as network input to generate corresponding actual image patches using a pre-trained neural network.

[0103] The stitching unit is used to stitch the actual image blocks according to the positions of the original images to obtain the SAR image.

[0104] Optionally, in one embodiment of the present application, the processing unit includes: a second processing subunit and a third processing subunit.

[0105] The second processing subunit is used to perform overall standardization on the simulated-actual image pair data set and remove outliers in the simulated-actual image pair data set to obtain a preliminary processed image.

[0106] The third processing subunit is used to normalize the initially processed image to obtain a processed image.

[0107] Optionally, in one embodiment of the present application, the generation module 300 further includes: a training unit and an optimization unit.

[0108] The training unit is used to train the neural network using the sample simulated SAR image data set and the corresponding sample SAR image data set, and calculate the corresponding loss function.

[0109] The optimization unit is used to optimize the neural network based on the loss function until the neural network meets the preset convergence condition and obtains the pre-trained neural network.

[0110] It should be noted that the above explanation of the data-physics dual-driven SAR image generation method embodiment is also applicable to the data-physics dual-driven SAR image generation device of this embodiment, and will not be repeated here.

[0111] According to the data-physics dual-driven SAR image generation device proposed in the embodiment of the present application, the target three-dimensional model can be directly obtained through the collected visible light information and the corresponding material information can be automatically assigned through the panoramic segmentation method, which greatly reduces the cost of manual modeling and material annotation. In terms of physics, a method with high computational efficiency is used to maintain the accuracy of the main structure of the generated image. In terms of data, a neural network is used to learn the difference between physical simulation images and real images, emphasizing fine structures and image noise to obtain more realistic SAR images. As a result, the technical problem in the related art that the existing physical or simulated scene-driven SAR image generation model has poor generalization ability when facing actual scenes, thereby limiting the development of deep learning in SAR image applications is solved.

[0112] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0113] A memory 701 , a processor 702 , and a computer program stored in the memory 701 and executable on the processor 702 .

[0114] When the processor 702 executes the program, the data-physics dual-driven SAR image generation method provided in the above embodiment is implemented.

[0115] Furthermore, the electronic device further comprises:

[0116] The communication interface 703 is used for communication between the memory 701 and the processor 702 .

[0117] The memory 701 is used to store computer programs that can be executed on the processor 702 .

[0118] The memory 701 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0119] If the memory 701, the processor 702 and the communication interface 703 are implemented independently, the communication interface 703, the memory 701 and the processor 702 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0120] Optionally, in a specific implementation, if the memory 701, the processor 702 and the communication interface 703 are integrated on a chip, the memory 701, the processor 702 and the communication interface 703 can communicate with each other through an internal interface.

[0121] The processor 702 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0122] This embodiment also provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the above data-physics dual-driven SAR image generation method is implemented.

[0123] The embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the data-physics dual-driven SAR image generation method provided in the embodiment of the present invention.

[0124] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0125] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0126] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0127] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0128] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0129] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0130] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0131] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A data-physics dual-driven SAR image generation method, characterized in that: The following steps are involved: Acquire SAR image sequences and visible light image sequences that meet preset conditions; Performing three-dimensional reconstruction on the visible light image sequence to obtain an initial three-dimensional grid, and assigning corresponding material parameters to the initial three-dimensional grid to obtain an actual three-dimensional grid with material information; The process of SAR finite physical modeling is simulated for the actual three-dimensional grid according to the parameters corresponding to each image in the image sequence to obtain a physical simulation image sequence, and a corresponding SAR image is generated based on the physical simulation image sequence and the image sequence.

2. The method according to claim 1, characterized in that The step of assigning corresponding material parameters to the initial three-dimensional mesh to obtain an actual three-dimensional mesh having material information includes: Performing panoptic segmentation on the initial three-dimensional grid to obtain corresponding segmentation results; The segmentation result is used to assign corresponding material parameters to the initial three-dimensional mesh to obtain the actual three-dimensional mesh.

3. The method according to claim 2, characterized in that The panoptic segmentation of the initial three-dimensional grid to obtain a corresponding segmentation result includes: Performing point cloud conversion on the initial three-dimensional grid to obtain a corresponding point cloud set; The point cloud set is panoramically segmented using a pre-trained point cloud segmentation model to obtain the segmentation result.

4. The method according to claim 1, characterized in that: The generating corresponding SAR images based on the physical simulation image sequence and the image sequence comprises: constructing a simulation-actual image pair data set using the physical simulation image sequence and the image sequence; Performing data processing on the simulation-actual image pair data set to obtain a processed image; Dividing the processed image into a plurality of image blocks using a preset division principle; Using the plurality of image blocks as network input to generate corresponding actual image blocks using a pre-trained neural network; The actual image blocks are spliced ​​according to the positions of the original images to obtain the SAR image.

5. The method according to claim 4, characterized in that The step of processing the simulation-actual image pair data set to obtain a processed image includes: Standardizing the simulated-actual image pair data set as a whole, and removing outliers in the simulated-actual image pair data set to obtain a preliminary processed image; The initially processed image is normalized to obtain the processed image.

6. The method according to claim 4, characterized in that Before using the pre-trained neural network to generate the corresponding actual image patches, it also includes: The neural network is trained using the sample simulated SAR image data set and the corresponding sample SAR image data set, and the corresponding loss function is calculated; The neural network is optimized based on the loss function until the neural network meets a preset convergence condition, and the pre-trained neural network is obtained.

7. A data-physics dual-driven SAR image generation device, characterized in that: include: An acquisition module, used to acquire a SAR image sequence and a visible light image sequence that meet preset conditions; A reconstruction module, used for performing three-dimensional reconstruction on the visible light image sequence to obtain an initial three-dimensional mesh, and assigning corresponding material parameters to the initial three-dimensional mesh to obtain an actual three-dimensional mesh with material information; A generation module is used to simulate the process of finite physical modeling of SAR on the actual three-dimensional grid according to the parameters corresponding to each image in the image sequence to obtain a physical simulation image sequence, and generate corresponding SAR images based on the physical simulation image sequence and the image sequence.

8. The device according to claim 7, characterized in that The reconstruction module comprises: A segmentation unit, used for performing panorama segmentation on the initial three-dimensional grid to obtain a corresponding segmentation result; The assigning unit is used to assign corresponding material parameters to the initial three-dimensional mesh using the segmentation result to obtain the actual three-dimensional mesh.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the data-physics dual-driven SAR image generation method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the data-physics dual-driven SAR image generation method as described in any one of claims 1 to 6.

Images