Neural network three-dimensional reconstruction method and device based on ISAR image sequence

Abstract

The application relates to a neural network three-dimensional reconstruction method and device based on an ISAR image sequence. The method comprises the following steps: representing a target based on an ISAR sequence image as a six-position function of a spatial position coordinate and an observation direction, and implicitly representing the six-position function by using a multilayer perceptron network; and training the network under the supervision of a two-dimensional ISAR image sequence without the three-dimensional structure of the target. By constructing all point coordinates in space, the three-dimensional voxel density of the target is obtained by using an optimized network, and the Marching Cube algorithm is further used to convert the voxel density into the three-dimensional surface element structure of the target to realize the reconstruction of the three-dimensional structure. By using the method, more complete and more accurate surface element structure information of a complex space target can be finally generated without feature extraction and correlation.

Description

Technical Field

[0001] This application relates to the field of radar three-dimensional imaging technology, and in particular to a neural network three-dimensional reconstruction method and apparatus based on ISAR image sequences. Background Technology

[0002] Inverse Synthetic Aperture Radar (ISAR) can acquire high-resolution radar images of moving targets around the clock and in all weather conditions, and is widely used in fields such as space surveillance, target identification, and radar astronomy. It achieves high radial resolution by transmitting a wide-bandwidth signal and obtains high lateral range resolution by using long-term observations to acquire a large accumulation angle. However, as the target moves, the two-dimensional imaging plane is constantly changing, making it difficult to capture the target's stable characteristics. If the target's three-dimensional information can be acquired, more comprehensive and stable information such as its spatial structure, size, and morphology can be extracted, allowing for further analysis of the target's motion attitude and prediction of its future actions and intentions. This increases our understanding of space target missions and provides strong support for target identification and target status assessment.

[0003] In recent years, there have been two main research approaches in ISAR 3D imaging: The first approach utilizes Interferometric Inverse Synthetic Aperture Radar (InISAR) to acquire multi-channel data containing path differences, then performs interferometric processing on this data to obtain the 3D coordinates of the target's scattering points, achieving 3D reconstruction of the target. However, this method has a complex overall system structure and is difficult to implement, resulting in limited practical applications. The second approach employs a time-for-space strategy, using a single-pulse radar to observe the target for extended periods, obtaining ISAR image sequences with significant attitude changes. Then, scattering points are extracted and correlated from the sequence to construct the target's observation matrix. Finally, a reconstruction algorithm is used to decompose the target's 3D point cloud from the observation matrix. However, this method has certain requirements regarding the attitude of the ISAR image sequence. How to fully utilize ISAR image sequences of the target under different attitudes and improve the accuracy of scattering point correlation has always been a key and challenging problem to be solved in this approach. This method can fully utilize the characteristics of existing monostation radars, such as low system design complexity and low cost.

[0004] In summary, with the development of radar technology, the application scenarios of radar imaging technology are constantly expanding. The demand for ISAR three-dimensional imaging technology of space targets is becoming increasingly urgent. Therefore, research on three-dimensional reconstruction technology of space targets based on ISAR image sequences can lay a solid technical foundation for my country's space situational awareness, space security, and space debris monitoring, and has significant theoretical research value and practical engineering application value. Summary of the Invention

[0005] Therefore, it is necessary to provide a neural network 3D reconstruction method and apparatus based on ISAR image sequences that can generate more complete and accurate surface structure information for complex spatial targets without the need for feature extraction and correlation, in order to address the above-mentioned technical problems.

[0006] A neural network-based 3D reconstruction method based on ISAR image sequences, the method comprising:

[0007] Acquire ISAR training data, which includes multiple frames of ISAR training images for a target arranged in chronological order, as well as the imaging plane vector and calibration value corresponding to each ISAR training image.

[0008] A six-dimensional function representing the spatial location of the target in each ISAR training image is constructed based on the corresponding imaging plane vector and calibration value of each ISAR training image. The six-dimensional function represents the spatial location coordinates and the observation direction.

[0009] The six-dimensional function corresponding to each ISAR training image is input into the MLP neural network, and the voxel density and three-dimensional complex scattering coefficients corresponding to the input six-dimensional function are output.

[0010] An ISAR rendered image is obtained by reconstructing based on the voxel density and three-dimensional complex scattering coefficients, and a loss function is constructed based on the ISAR rendered image and the corresponding ISAR training image.

[0011] The MLP neural network is trained according to the loss function to obtain a trained MLP neural network;

[0012] Obtain the ISAR image sequence of the target to be reconstructed in three dimensions, as well as the imaging plane vector and calibration value corresponding to each ISAR image in the ISAR image sequence, and construct the corresponding six-dimensional function based on the imaging plane vector and calibration value;

[0013] The six-dimensional function corresponding to each ISAR image is input into the trained MLP neural network, which outputs the corresponding voxel density and three-dimensional complex scattering coefficient. The voxel density is converted into a three-dimensional surface model of the target to be reconstructed according to the Marching Cube algorithm to achieve three-dimensional reconstruction.

[0014] In one embodiment, the method further includes:

[0015] Obtain a new observation direction for the target to be reconstructed in three dimensions, replace the observation direction in the six-dimensional function with the new observation direction, and input the six-dimensional function into the trained MLP neural network to output the corresponding voxel density and three-dimensional complex scattering coefficients;

[0016] Based on the three-dimensional complex scattering coefficients, construct an ISAR rendered image of the target to be reconstructed in three dimensions under a new observation direction.

[0017] In one embodiment, constructing the spatial location coordinates of the target in the corresponding ISAR training image based on the imaging plane vector and calibration value of each ISAR training image includes:

[0018] The imaging plane vector includes a direction dimension vector and a distance dimension vector;

[0019] Based on the fact that the origin of the ISAR training image center is consistent with the origin of the target body coordinate system, the specified points on the ISAR training image in the pixel coordinate system are transformed to be represented in three-dimensional coordinates in the target body system according to the imaging plane vector and calibration value;

[0020] Then, the coordinate values ​​of the sampling point along the normal of the imaging plane at the specified point in space are constructed to obtain the spatial location coordinates of the target in the ISAR training image.

[0021] In one embodiment, the transformation of a specified point on the ISAR training image in the pixel coordinate system to a three-dimensional coordinate system in the target local system based on the imaging plane vector and calibration value is performed using the following formula:

[0022]

[0023] In the above formula, and These represent the x-coordinate and y-coordinate of the specified point in the pixel coordinate system, respectively. and These represent the calibration values ​​for the azimuth and distance dimensions, respectively. and These represent the distance dimension vector and the azimuth dimension vector, respectively. and These represent the height and width of the ISAR training image, respectively, in pixels.

[0024] In one embodiment, the coordinates of the sampling point at a specified point in the reconstructed space along the normal to the imaging plane are expressed by the following formula:

[0025]

[0026] In the above formula, This represents the three-dimensional coordinates of the specified point in the target system, where N is the number of sampling points. Indicates the normal to the imaging plane. and These represent the starting and ending points of the integration along the normal to the imaging plane, respectively.

[0027] In one embodiment, constructing the observation direction of the target space in the corresponding ISAR training image based on the corresponding imaging plane vector and calibration value of each ISAR training image includes:

[0028] The distance vector and the imaging plane normal are combined to form the observation direction.

[0029] In one embodiment, the step of inputting the six-dimensional function corresponding to each ISAR training image into the MLP neural network and outputting the voxel density and three-dimensional complex scattering coefficients corresponding to the input six-dimensional function includes:

[0030] The MLP neural network includes a first part and a second part, wherein the first part includes eight fully connected layers using the ReLU activation function, and the second part includes a fully connected network and the Tanh activation function;

[0031] Inputting the spatial location coordinates into the first part yields the voxel density and multidimensional features of the spatial points;

[0032] The multidimensional features and the observation direction are input into the second part to obtain the three-dimensional complex scattering coefficients.

[0033] In one embodiment, the step of reconstructing the ISAR rendered image based on the voxel density and three-dimensional complex scattering coefficients includes:

[0034] The expected value of the three-dimensional complex scattering coefficients of the sampling points is calculated along the normal of the imaging plane, and the expected value is combined into a complex image, which is then used as the ISAR rendered image.

[0035] In one embodiment, the loss function is the mean square error between the magnitude of the ISAR rendered image and the magnitude of the corresponding ISAR training image.

[0036] A neural network-based 3D reconstruction device based on ISAR image sequences, the device comprising:

[0037] The training data acquisition module is used to acquire ISAR training data, which includes multiple frames of ISAR training images for a target arranged in chronological order, as well as the imaging plane vector and calibration value corresponding to each ISAR training image.

[0038] The six-dimensional function construction module is used to construct a six-dimensional function of the target space in the corresponding ISAR training image based on the imaging plane vector and calibration value of each ISAR training image. The six-dimensional function is the spatial position coordinates and observation direction.

[0039] The voxel density and three-dimensional complex scattering coefficient acquisition module is used to input the six-dimensional function corresponding to each ISAR training image into the MLP neural network, and output the voxel density and three-dimensional complex scattering coefficient corresponding to the input six-dimensional function.

[0040] The loss function construction module is used to reconstruct an ISAR rendered image based on the three-dimensional complex scattering coefficients, and to construct a loss function based on the ISAR rendered image and the corresponding ISAR training image.

[0041] The MLP neural network training module trains the MLP neural network according to the loss function to obtain a trained MLP neural network.

[0042] The six-dimensional function construction module for the target to be reconstructed in three dimensions is used to obtain the ISAR image sequence of the target to be reconstructed in three dimensions, as well as the imaging plane vector and calibration value corresponding to each ISAR image in the ISAR image sequence, and to construct the corresponding six-dimensional function based on the imaging plane vector and calibration value.

[0043] The 3D reconstruction module is used to input the six-dimensional function corresponding to each ISAR image into the trained MLP neural network, output the corresponding voxel density and 3D complex scattering coefficient, and convert the voxel density into a 3D surface model of the target to be reconstructed according to the Marching Cube algorithm to achieve 3D reconstruction.

[0044] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0045] Acquire ISAR training data, which includes multiple frames of ISAR training images for a target arranged in chronological order, as well as the imaging plane vector and calibration value corresponding to each ISAR training image.

[0046] A six-dimensional function representing the spatial location of the target in each ISAR training image is constructed based on the corresponding imaging plane vector and calibration value of each ISAR training image. The six-dimensional function represents the spatial location coordinates and the observation direction.

[0047] The six-dimensional function corresponding to each ISAR training image is input into the MLP neural network, and the voxel density and three-dimensional complex scattering coefficients corresponding to the input six-dimensional function are output.

[0048] An ISAR rendered image is obtained by reconstructing based on the voxel density and three-dimensional complex scattering coefficients, and a loss function is constructed based on the ISAR rendered image and the corresponding ISAR training image.

[0049] The MLP neural network is trained according to the loss function to obtain a trained MLP neural network;

[0050] Obtain the ISAR image sequence of the target to be reconstructed in three dimensions, as well as the imaging plane vector and calibration value corresponding to each ISAR image in the ISAR image sequence, and construct the corresponding six-dimensional function based on the imaging plane vector and calibration value;

[0051] The six-dimensional function corresponding to each ISAR image is input into the trained MLP neural network, which outputs the corresponding voxel density and three-dimensional complex scattering coefficient. The voxel density is converted into a three-dimensional surface model of the target to be reconstructed according to the Marching Cube algorithm to achieve three-dimensional reconstruction.

[0052] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0053] Acquire ISAR training data, which includes multiple frames of ISAR training images for a target arranged in chronological order, as well as the imaging plane vector and calibration value corresponding to each ISAR training image.

[0054] A six-dimensional function representing the spatial location of the target in each ISAR training image is constructed based on the corresponding imaging plane vector and calibration value of each ISAR training image. The six-dimensional function represents the spatial location coordinates and the observation direction.

[0055] The six-dimensional function corresponding to each ISAR training image is input into the MLP neural network, and the voxel density and three-dimensional complex scattering coefficients corresponding to the input six-dimensional function are output.

[0056] An ISAR rendered image is obtained by reconstructing based on the voxel density and three-dimensional complex scattering coefficients, and a loss function is constructed based on the ISAR rendered image and the corresponding ISAR training image.

[0057] The MLP neural network is trained according to the loss function to obtain a trained MLP neural network;

[0058] Obtain the ISAR image sequence of the target to be reconstructed in three dimensions, as well as the imaging plane vector and calibration value corresponding to each ISAR image in the ISAR image sequence, and construct the corresponding six-dimensional function based on the imaging plane vector and calibration value;

[0059] The six-dimensional function corresponding to each ISAR image is input into the trained MLP neural network, which outputs the corresponding voxel density and three-dimensional complex scattering coefficient. The voxel density is converted into a three-dimensional surface model of the target to be reconstructed according to the Marching Cube algorithm to achieve three-dimensional reconstruction.

[0060] The aforementioned neural network-based 3D reconstruction method and apparatus based on ISAR image sequences represent the target as a six-dimensional function of spatial location coordinates and observation direction using ISAR image sequences. This representation is implicitly performed using a Multilayer Perceptron (MLP) network. Even without a 3D target structure, the network is trained using only 2D ISAR image sequences under supervision. By constructing the coordinates of all points in space, the optimized network is used to obtain the 3D voxel density of the target. Furthermore, the Marching Cube algorithm is used to transform the voxel density into a surface structure of the target. This method, without requiring feature extraction and correlation, ultimately generates more complete and accurate surface structure information for complex spatial targets. Attached Figure Description

[0061] Figure 1 This is a flowchart illustrating a neural network-based 3D reconstruction method based on ISAR image sequences in one embodiment.

[0062] Figure 2 This is a schematic diagram of the radar nerve radiation field in one embodiment;

[0063] Figure 3 This is a schematic diagram of the structure of an MLP network in one embodiment;

[0064] Figure 4 This is a schematic diagram illustrating the acquisition of spatial coordinates and observation direction in one embodiment;

[0065] Figure 5 A visualization of three scattering point models in a simulation experiment;

[0066] Figure 6 This is a schematic diagram of a simulated ISAR image sequence for three scattering point models in a simulation experiment.

[0067] Figure 7 This is a schematic diagram comparing the reconstruction results of three scattering point models with the true values ​​in a simulation experiment.

[0068] Figure 8 This is a detailed schematic diagram of the reconstruction using this method in a simulation experiment;

[0069] Figure 9 This is a schematic diagram of the rendering and simulation of ISAR images of three scattering point models in a new observation direction during a simulation experiment.

[0070] Figure 10 This is a structural block diagram of a neural network 3D reconstruction device based on ISAR image sequences in one embodiment;

[0071] Figure 11 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0073] like Figure 1 As shown, a neural network-based 3D reconstruction method based on ISAR image sequences is provided, including the following steps:

[0074] Step S100: Obtain ISAR training data, which includes multiple frames of ISAR training images for a target arranged in chronological order, as well as the imaging plane vector and calibration value corresponding to each ISAR training image.

[0075] Step S110: Construct a six-dimensional function of the target space in the corresponding ISAR training image based on the imaging plane vector and calibration value of each ISAR training image. The six-dimensional function is the spatial location coordinates and observation direction.

[0076] Step S120: Input the six-dimensional function corresponding to each ISAR training image into the MLP neural network, and output the voxel density and three-dimensional complex scattering coefficients corresponding to the input six-dimensional function.

[0077] Step S130: Reconstruct the ISAR rendered image based on the three-dimensional complex scattering coefficients, and construct a loss function based on the ISAR rendered image and the corresponding ISAR training image.

[0078] Step S140: Train the MLP neural network according to the loss function to obtain the trained MLP neural network;

[0079] Step S150: Obtain the ISAR image sequence of the target to be reconstructed in three dimensions, as well as the imaging plane vector and calibration value corresponding to each ISAR image in the ISAR image sequence, and construct the corresponding six-dimensional function based on the imaging plane vector and calibration value.

[0080] Step S160: Input the six-dimensional function corresponding to each ISAR image into the trained MLP neural network, output the corresponding voxel density and three-dimensional complex scattering coefficient, and convert the voxel density into a three-dimensional surface model of the target to be reconstructed according to the Marching Cube algorithm to achieve three-dimensional reconstruction.

[0081] This method addresses the challenges of feature extraction and correlation encountered in traditional 3D reconstruction methods by proposing a neural network-based 3D reconstruction method based on ISAR image sequences, named Radar Neural Radiance Fields (RaNeRF). Figure 2 As shown, this method represents a spatial target as a six-dimensional function of its spatial location and observation direction, implicitly represented by a Multilayer Perceptron (MLP) network. Without the target's three-dimensional structure, the network is trained using only two-dimensional ISAR image sequences under supervision. By constructing the coordinates of all points in space, the optimized network is used to obtain the target's three-dimensional voxel density. Furthermore, the Marching Cube algorithm is used to transform the voxel density into the target's surface structure. This method, without requiring feature extraction and correlation, ultimately generates more complete and accurate surface structure information for complex spatial targets. Moreover, it can be used to generate ISAR images from any observation direction.

[0082] It should be noted that steps S100 to S140 are the part for training the MLP neural network, and steps S150 to S160 are the part for applying the trained MLP neural network in practice.

[0083] In this embodiment, the target of this method is a space target, including satellites, aircraft, etc.

[0084] In step S100, ISAR training images for a single target object are acquired. These training images can be obtained from simulation experiments or from radar detection of real targets. All ISAR training images are complex images.

[0085] In this embodiment, the target's three-dimensional structure is mapped to a six-dimensional function, that is, the target's three-dimensional structure is represented as the spatial coordinates of the target's location. and observation direction The observation direction includes the radar line-of-sight direction (range vector) and its corresponding imaging plane normal, which serve as inputs to the MLP neural network and output the voxel density of spatial points. and the real and imaginary parts of the three-dimensional complex scattering coefficients For example, in step S120.

[0086] In this embodiment, an MLP (Multilayer Perceptron, MLP) network is used to implicitly represent a six-dimensional function. This MLP neural network includes a first part and a second part, as follows: Figure 3 As shown in the diagram, the number in each square represents the dimension of the network. The first part includes eight fully connected layers using the ReLU activation function, and the second part includes a fully connected network and the Tanh activation function. Its structure is as follows. Figure 3 As shown. The spatial location coordinates are input into the first part to obtain the voxel density and multidimensional features of the spatial point. Then, the multidimensional features and the observation direction are input into the second part to obtain the three-dimensional complex scattering coefficients.

[0087] Specifically, the first part of the MLP neural network consists of eight fully connected layers using the ReLU activation function, each with 256 channels. The input to this part of the network... Output voxel density The extracted 256-dimensional features are then input into another 128-dimensional fully connected network along with the observation direction, and the corresponding three-dimensional complex scattering coefficients are output using the Tanh activation function.

[0088] In step S110, to facilitate the construction of the input for the MLP neural network, under the target's own system, the radar line-of-sight direction and its corresponding imaging plane normal are represented by unit three-dimensional vectors, and then concatenated into a six-dimensional vector as the observation direction. At this point, the MLP neural network... The approximate three-dimensional structure representation of the above continuous six-dimensional target is obtained by optimizing the corresponding weights. We obtain the voxel density and three-dimensional complex scattering coefficients corresponding to the six-dimensional input.

[0089] To facilitate the construction of the input to the MLP neural network, namely the coordinates of the sampling points in space and the corresponding observation direction (i.e., a six-dimensional function), this method combines the estimated ISAR imaging plane vector and calibration value to propose a method for calculating the spatial location coordinates from the pixel coordinates of the ISAR image:

[0090] like Figure 4 As shown, the spatial coordinates of the target in the corresponding ISAR training image are constructed based on the imaging plane vector and calibration value of each ISAR training image. The imaging plane vector includes both a direction vector and a range vector. With the origin of the ISAR training image center aligned with the origin of the target's body coordinate system, a specified point in the ISAR training image in the pixel coordinate system is transformed to represent its position in the three-dimensional coordinates of the target's body coordinate system based on the imaging plane vector and calibration value. Then, the coordinates of the sampling point along the normal to the imaging plane at the specified point in space are constructed to obtain the spatial coordinates of the target in the ISAR training image.

[0091] Specifically, it is assumed that the origin of the center of the ISAR complex image (i.e., the ISAR training image) is consistent with the origin of the target body coordinate system. This assumption can be achieved through center offset correction.

[0092] Let a point on the ISAR complex image in the pixel coordinate system be... With the upper left corner of the ISAR complex image as the origin, then The three-dimensional coordinates of a point in the target system It can be represented as:

[0093] (1)

[0094] In formula (1), and These represent the x-coordinate and y-coordinate of the specified point in the pixel coordinate system, respectively. and These represent the calibration values ​​for the azimuth and distance dimensions, respectively. and These represent the distance dimension vector and the azimuth dimension vector, respectively. and The height and width of the ISAR training image are represented by pixels, respectively. The horizontal and vertical axis data correspond to the azimuth and distance dimensions, respectively.

[0095] This allows us to construct space The coordinates of the sampling point along the normal to the imaging plane at the given point:

[0096] (2)

[0097] In the above formula, This represents the three-dimensional coordinates of a specified point within the target system, where N is the number of sampling points. Indicates the normal to the imaging plane. and These represent the starting and ending points of the integration along the normal to the imaging plane, respectively.

[0098] In this embodiment, the observation direction of the target space in the corresponding ISAR training image is constructed based on the imaging plane vector and calibration value of each ISAR training image, including: stitching the distance dimension vector and the imaging plane normal to form the observation direction.

[0099] In step S130, reconstructing the ISAR rendered image based on the three-dimensional complex scattering coefficients includes: calculating the expected value of the three-dimensional complex scattering coefficients of the sampling points along the normal of the imaging plane, combining the expected values ​​into a complex image, and using the complex image as the ISAR rendered image.

[0100] Specifically, according to the ISAR image projection principle, points on a ray along the normal of the imaging plane will be projected onto the same location on the image. Therefore, this method calculates the expected value of the three-dimensional complex scattering coefficients of the sampling points along the normal of the imaging plane based on the output of the MLP network, and combines them into a complex image as the ISAR rendered image.

[0101] Assuming along the normal of the imaging plane The rays are: Therefore, the expected value of the real and imaginary parts of the ISAR training image is:

[0102] (3)

[0103] In formula (3), To obtain the 3D coordinates of each point in the ISAR training image in the target's body coordinate system. and These are the lower limit (starting point) and upper limit (ending point) of the integral along the normal to the imaging plane, respectively. Function Indicates that along the normal of the imaging plane from arrive The cumulative voxel density is the probability that radar rays are not reflected.

[0104] Rendering new ISAR training images requires estimating the integral of each pixel along the normal to the imaging plane. The continuous form of equation (3) can be replaced by a discrete integral:

[0105] (4)

[0106] In formula (4), This represents the distance between adjacent sampling points along the normal to the imaging plane.

[0107] To effectively sample and render high-resolution complex spatial scenes, a positional encoding and hierarchical sampling process is set up. The encoding function is as follows:

[0108] (5)

[0109] In formula (5), This represents any value in the vector input to the MLP network. This indicates that the input will be taken from space. Mapping to higher-dimensional space During network training, the original numerical value and its corresponding encoded value are concatenated and input into the network. For spatial location... ,have Regarding the direction of observation have .

[0110] The stratified sampling process includes coarse sampling and fine sampling. The number of sampling points in coarse sampling... At this point, formula (4) can be rewritten as:

[0111] (6)

[0112] Calculate weights based on the coarsely sampled network output. Determine the number of fine samples The position coordinates will The coordinates of each sampling point are input into the network. Using formula (4) and the network output, the real and imaginary parts of the complex data of the ISAR training image under fine sampling are calculated. .

[0113] In step S130, a loss function is constructed based on the mean square error between the magnitude of the ISAR rendered image and the magnitude of the corresponding ISAR training image. The loss function is expressed as follows:

[0114] (7)

[0115] In formula (7), This represents the number of segmented rays set. For rays The set of rays Down, and These are the real and imaginary ground truth values ​​of the corresponding ISAR training images, respectively. These are the real part output and imaginary part output of coarse mesh rendering, respectively. These are the real and imaginary outputs of the fine-mesh rendering, respectively.

[0116] Finally, when using the trained MLP neural network for 3D reconstruction, the six-dimensional function corresponding to the ISAR image sequence of the target to be reconstructed is input into the trained MLP neural network, which outputs the corresponding voxel density and 3D complex scattering coefficients. The voxel density is then converted into a 3D surface model of the target to be reconstructed using the Marching Cube algorithm to achieve 3D reconstruction.

[0117] Furthermore, this method can also construct arbitrary observation directions given a standard value and input them into a trained MLP neural network to render ISAR images under that observation direction, thus enabling the generation of ISAR images under arbitrary observation directions.

[0118] In this paper, simulation experiments are also conducted to demonstrate the effectiveness of the proposed method.

[0119] like Figure 5The image shows three scattering point models used in the simulation. The simple scattering point model has 65 scattering points and a relatively simple structure. Figure 5 (a1)- Figure 5 (a4) indicates that the satellite I scattering point model has a total of 310,022 scattering points and a more complex structure, including two sections of the cabin and a pair of solar panels. Figure 5 (b1)- Figure 5 (b4) indicates that the Satellite II scattering point model has 631,479 scattering points and a more complex structure, including three cabin sections and two pairs of mutually perpendicular solar panels. Figure 5 (c1)- Figure 5 (c4) indicates that, among which Figure 5 (a1) Figure 5 (b1), and Figure 5 (c1) shows their projection onto the O-XY plane. Figure 5 (a2) Figure 5 (b2), and Figure 5 (c2) shows their projection onto the O-YZ plane. Figure 5 (a3) Figure 5 (b3), and Figure 5 (c3) shows their projection onto the O-XZ plane. Figure 5 (a4) Figure 5 (b4), and Figure 5 (c4) shows their oblique view in the O-XZ plane.

[0120] like Figure 6 The image shown is a simulated ISAR image sequence for three scattering point models. Figure 6 (a1) to Figure 6 (a3) represents a simple scattering point model. Figure 6 (b1) to Figure 6 (b3) represents satellite I. Figure 6 (c1) to Figure 6 (c3) represents satellite II, where, Figure 6 (a1) Figure 6 (b1) and Figure 6 (c1) The first frame of ISAR image obtained by simulation using the RD algorithm with three different scattering point models. Figure 6 (a2) Figure 6 (b2) and Figure 6 (c2) are the 15th frame ISAR images obtained by simulation using the RD algorithm with three scattering point models. Figure 6 (a3) Figure 6 (b3) and Figure 6(c3) shows the 100th frame of ISAR image obtained by simulation using the RD algorithm with three scattering point models. It can be seen that the focusing effect is good, and the contrast in scattering point intensity between satellites I and II is more obvious, which is beneficial to network training and can support the correctness of the reconstruction results. The specific simulation parameters are set as follows: initially, the radar operates in the Ku band with a center frequency bandwidth of 1 GHz. For the training dataset, the azimuth angle of LOS is fixed at [value missing]. The set of pitch angles is For the test dataset, the azimuth angle of LOS is set to a fixed value. The set of pitch angles is Each time, a pitch angle is selected, and the rotation axis is a fixed axis with an angular velocity of... The total observation time was Finally, ISAR complex images were obtained using the traditional RD algorithm. The final training data contained 644 images, and the test data contained 139 images.

[0121] like Figure 7 As shown, the reconstruction results of three scattering point models are compared with the true values. Figure 7 (a1) to Figure 7 (a4) represents a simple scattering point model. Figure 7 (b1) to Figure 7 (b4) represents satellite I. Figure 7 (c1) to Figure 7 (c4) represents satellite II, where, Figure 7 (a1) to Figure 7 (a4) shows a comparison of the 3D reconstruction results of the traditional method and the present method with the true value of the target on the projection and oblique view of the simple scattering point on the O-XY plane, O-YZ plane, and O-XZ plane. Figure 7 (b1) to Figure 7 (b4) Comparison of the three-dimensional reconstruction results of satellite I on the projection and oblique view of the O-XY plane, O-YZ plane, and O-XZ plane, using traditional methods and the present method, with the true value of the target. Figure 7 (c1) to Figure 7 (c4) Comparison of the 3D reconstruction results of the traditional method and the proposed method with the true value of the target on the projection and oblique view of Satellite II in the O-XY plane, O-YZ plane, and O-XZ plane. To distinguish the results of the three methods, the Z value of the traditional method is shifted upward by 10m, and the Z value of the true value is shifted downward by 10m. As can be seen from the figure, whether it is the simple scattering point model or the more complex Satellite I and Satellite II models, compared with the point cloud 1 of the traditional method, the three-dimensional structural information of the proposed method is richer, the structure is more complete and accurate, and the overlap with the true value is higher.

[0122] like Figure 8The image shows a detailed demonstration of the reconstruction results using this method. It can be seen that... Figure 8 (a) The two sections of the Satellite I model can be effectively restored. Figure 8 (b) Satellite II’s two pairs of solar panels remain vertical, and the three sections of the cabin are also clearly defined in terms of detail, giving it a strong sense of three-dimensionality.

[0123] like Figure 9 The image shown is an ISAR image rendered and simulated using three different scattering point models in the new observation direction. Figure 9 (a1) to Figure 9 (a6) represents a simple scattering point model. Figure 9 (b1) to Figure 9 (b6) represents satellite I. Figure 9 (c1) to Figure 9 (c6) represents Satellite II, where Figure 9 (a1) Figure 9 (b1) Figure 9 (c1) Figure 9 (a3) Figure 9 (b3) Figure 9 (c3), 9 (a5), Figure 9 (b5) Figure 9 (c5) shows rendered ISAR images with radar line-of-sight directions of (-0.28, -0.41, 0.87), (0.19, 0.46, 0.87), and (-0.68, 0.53, 0.50), and corresponding imaging plane normals of (0.49, 0.71, 0.50), (-0.33, -0.80, 0.50), and (0.39, -0.31, 0.87), respectively. Figure 9 (a2) Figure 9 (b2) Figure 9 (c2) Figure 9 (a4) Figure 9 (b4) Figure 9 (c4), 9(a6) Figure 9 (b6) Figure 9 (c6) shows the corresponding simulated ISAR image. As can be seen from the figure, the ISAR image rendered by this method has roughly the same shape as the simulated ISAR image, thus it can be used to generate ISAR images under any observation direction, and further used for tasks such as data augmentation and target recognition.

[0124] In summary, this method can directly reconstruct the 3D surface structure of a target without the need for feature extraction and correlation of radar image sequences. Compared with the 3D point clouds obtained by traditional methods, the structural information is more complete and accurate. Furthermore, the optimized model can generate ISAR images from any observation direction, which can be used for tasks such as data augmentation and target recognition. It has significant engineering application value for acquiring 3D structural information of space targets.

[0125] The aforementioned neural network-based 3D reconstruction method based on ISAR image sequences can directly reconstruct the 3D surface structure information of the target without the need for radar image sequence feature extraction and correlation. Compared with the 3D point cloud obtained by traditional methods, the structural information is more complete and accurate. Furthermore, the optimized model can generate ISAR images from any observation direction, which can be used for tasks such as data augmentation and target recognition. This method has significant engineering application value for acquiring 3D structural information of space targets.

[0126] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0127] In one embodiment, such as Figure 10 As shown, a neural network-based 3D reconstruction device based on ISAR image sequences is provided, comprising: a training data acquisition module 200, a six-dimensional function construction module 210, a voxel density and 3D complex scattering coefficient acquisition module 220, a loss function construction module 230, an MLP neural network training module 240, a six-dimensional function construction module 250 for the target to be reconstructed, and a 3D reconstruction module 260, wherein:

[0128] The training data acquisition module 200 is used to acquire ISAR training data, which includes multiple frames of ISAR training images for a target arranged in chronological order, as well as imaging plane vectors and calibration values ​​corresponding to each ISAR training image.

[0129] The six-dimensional function construction module 210 is used to construct a six-dimensional function of the target space in the corresponding ISAR training image based on the imaging plane vector and calibration value of each ISAR training image. The six-dimensional function is the spatial position coordinates and the observation direction.

[0130] The voxel density and three-dimensional complex scattering coefficient acquisition module 220 is used to input the six-dimensional function corresponding to each ISAR training image into the MLP neural network, and output the voxel density and three-dimensional complex scattering coefficient corresponding to the input six-dimensional function.

[0131] The loss function construction module 230 is used to reconstruct an ISAR rendered image based on the voxel density and three-dimensional complex scattering coefficients, and to construct a loss function based on the ISAR rendered image and the corresponding ISAR training image.

[0132] MLP neural network training module 240 trains the MLP neural network according to the loss function to obtain a trained MLP neural network;

[0133] The six-dimensional function construction module 250 for the target to be reconstructed in three dimensions is used to obtain the ISAR image sequence of the target to be reconstructed in three dimensions, as well as the imaging plane vector and calibration value corresponding to each ISAR image in the ISAR image sequence, and to construct the corresponding six-dimensional function based on the imaging plane vector and calibration value.

[0134] The 3D reconstruction module 260 is used to input the six-dimensional function corresponding to each of the ISAR images into the trained MLP neural network, output the corresponding voxel density and 3D complex scattering coefficients, and convert the voxel density into a 3D surface model of the target to be reconstructed according to the Marching Cube algorithm to achieve 3D reconstruction.

[0135] Specific limitations regarding the neural network 3D reconstruction device based on ISAR image sequences can be found in the limitations of the neural network 3D reconstruction method based on ISAR image sequences mentioned above, and will not be repeated here. Each module in the aforementioned neural network 3D reconstruction device based on ISAR image sequences can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0136] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 11As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a neural network-based 3D reconstruction method based on ISAR image sequences. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0137] Those skilled in the art will understand that Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0138] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0139] Acquire ISAR training data, which includes multiple frames of ISAR training images for a target arranged in chronological order, as well as the imaging plane vector and calibration value corresponding to each ISAR training image.

[0140] A six-dimensional function representing the spatial location of the target in each ISAR training image is constructed based on the corresponding imaging plane vector and calibration value of each ISAR training image. The six-dimensional function represents the spatial location coordinates and the observation direction.

[0141] The six-dimensional function corresponding to each ISAR training image is input into the MLP neural network, and the voxel density and three-dimensional complex scattering coefficients corresponding to the input six-dimensional function are output.

[0142] An ISAR rendered image is obtained by reconstructing based on the voxel density and three-dimensional complex scattering coefficients, and a loss function is constructed based on the ISAR rendered image and the corresponding ISAR training image.

[0143] The MLP neural network is trained according to the loss function to obtain a trained MLP neural network;

[0144] Obtain the ISAR image sequence of the target to be reconstructed in three dimensions, as well as the imaging plane vector and calibration value corresponding to each ISAR image in the ISAR image sequence, and construct the corresponding six-dimensional function based on the imaging plane vector and calibration value;

[0145] The six-dimensional function corresponding to each ISAR image is input into the trained MLP neural network, which outputs the corresponding voxel density and three-dimensional complex scattering coefficient. The voxel density is converted into a three-dimensional surface model of the target to be reconstructed according to the Marching Cube algorithm to achieve three-dimensional reconstruction.

[0146] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0147] Acquire ISAR training data, which includes multiple frames of ISAR training images for a target arranged in chronological order, as well as the imaging plane vector and calibration value corresponding to each ISAR training image.

[0148] A six-dimensional function representing the spatial location of the target in each ISAR training image is constructed based on the corresponding imaging plane vector and calibration value of each ISAR training image. The six-dimensional function represents the spatial location coordinates and the observation direction.

[0149] The six-dimensional function corresponding to each ISAR training image is input into the MLP neural network, and the voxel density and three-dimensional complex scattering coefficients corresponding to the input six-dimensional function are output.

[0150] An ISAR rendered image is obtained by reconstructing based on the voxel density and three-dimensional complex scattering coefficients, and a loss function is constructed based on the ISAR rendered image and the corresponding ISAR training image.

[0151] The MLP neural network is trained according to the loss function to obtain a trained MLP neural network;

[0152] Obtain the ISAR image sequence of the target to be reconstructed in three dimensions, as well as the imaging plane vector and calibration value corresponding to each ISAR image in the ISAR image sequence, and construct the corresponding six-dimensional function based on the imaging plane vector and calibration value;

[0153] The six-dimensional function corresponding to each ISAR image is input into the trained MLP neural network, which outputs the corresponding voxel density and three-dimensional complex scattering coefficient. The voxel density is converted into a three-dimensional surface model of the target to be reconstructed according to the Marching Cube algorithm to achieve three-dimensional reconstruction.

[0154] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0155] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0156] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A neural network-based 3D reconstruction method based on ISAR image sequences, characterized in that, The method includes: Acquire ISAR training data, which includes multiple frames of ISAR training images for a target arranged in chronological order, as well as the imaging plane vector and calibration value corresponding to each ISAR training image; A six-dimensional function representing the spatial location of the target in each ISAR training image is constructed based on the corresponding imaging plane vector and calibration value of each ISAR training image. The six-dimensional function represents the spatial location coordinates and the observation direction. The six-dimensional function corresponding to each ISAR training image is input into the MLP neural network, and the voxel density and three-dimensional complex scattering coefficients corresponding to the input six-dimensional function are output. An ISAR rendered image is obtained by reconstructing based on the voxel density and three-dimensional complex scattering coefficients, and a loss function is constructed based on the ISAR rendered image and the corresponding ISAR training image. The MLP neural network is trained according to the loss function to obtain a trained MLP neural network; Obtain the ISAR image sequence of the target to be reconstructed in three dimensions, as well as the imaging plane vector and calibration value corresponding to each ISAR image in the ISAR image sequence, and construct the corresponding six-dimensional function based on the imaging plane vector and calibration value; The six-dimensional function corresponding to each ISAR image is input into the trained MLP neural network, which outputs the corresponding voxel density and three-dimensional complex scattering coefficient. The voxel density is converted into a three-dimensional surface model of the target to be reconstructed according to the Marching Cube algorithm to achieve three-dimensional reconstruction. The method further includes: Obtain a new observation direction for the target to be reconstructed in three dimensions, replace the observation direction in the six-dimensional function with the new observation direction, and input the six-dimensional function into the trained MLP neural network to output the corresponding voxel density and three-dimensional complex scattering coefficients; Based on the voxel density and three-dimensional complex scattering coefficients, construct an ISAR rendered image of the target to be reconstructed in three dimensions under a new observation direction.

2. The neural network three-dimensional reconstruction method according to claim 1, characterized in that, The step of constructing the spatial coordinates of the target in the corresponding ISAR training image based on the imaging plane vector and calibration value of each ISAR training image includes: The imaging plane vector includes a direction dimension vector and a distance dimension vector; Based on the fact that the origin of the ISAR training image center is consistent with the origin of the target body coordinate system, the specified points on the ISAR training image in the pixel coordinate system are transformed to be represented in three-dimensional coordinates in the target body system according to the imaging plane vector and calibration value; Then, the coordinate values ​​of the sampling point along the normal of the imaging plane at the specified point in space are constructed to obtain the spatial location coordinates of the target in the ISAR training image.

3. The neural network three-dimensional reconstruction method according to claim 2, characterized in that, The process of transforming a specified point on the ISAR training image in the pixel coordinate system to a three-dimensional coordinate system in the target local system based on the imaging plane vector and calibration value uses the following formula: In the above formula, and These represent the x-coordinate and y-coordinate of the specified point in the pixel coordinate system, respectively. and These represent the calibration values ​​for the azimuth and distance dimensions, respectively. and These represent the distance dimension vector and the azimuth dimension vector, respectively. and These represent the height and width of the ISAR training image, respectively, in pixels.

4. The neural network three-dimensional reconstruction method according to claim 3, characterized in that, The coordinates of the sampling point along the normal to the imaging plane at a specified point in the reconstructed space are calculated using the following formula: In the above formula, This represents the three-dimensional coordinates of the specified point within the target system, where N is the number of sampling points. Indicates the normal to the imaging plane. and These represent the starting and ending points of the integration along the normal to the imaging plane, respectively.

5. The neural network three-dimensional reconstruction method according to claim 4, characterized in that, The process of constructing the observation direction of the target space in the corresponding ISAR training image based on the corresponding imaging plane vector and calibration value of each ISAR training image includes: The distance vector and the imaging plane normal are combined to form the observation direction.

6. The neural network three-dimensional reconstruction method according to claim 5, characterized in that, The step of inputting the six-dimensional function corresponding to each ISAR training image into the MLP neural network and outputting the voxel density and three-dimensional complex scattering coefficients corresponding to the input six-dimensional function includes: The MLP neural network includes a first part and a second part, wherein the first part includes eight fully connected layers using the ReLU activation function, and the second part includes a fully connected network and the Tanh activation function; Inputting the spatial location coordinates into the first part yields the voxel density and multidimensional features of the spatial points; The multidimensional features and the observation direction are input into the second part to obtain the three-dimensional complex scattering coefficients.

7. The neural network three-dimensional reconstruction method according to claim 6, characterized in that, The process of reconstructing the ISAR rendered image based on the voxel density and three-dimensional complex scattering coefficients includes: The expected value of the three-dimensional complex scattering coefficients of the sampling points is calculated along the normal of the imaging plane, and the expected value is combined into a complex image, which is then used as the ISAR rendered image.

8. The neural network three-dimensional reconstruction method according to claim 6, characterized in that, The loss function is the mean square error between the magnitude of the ISAR rendered image and the magnitude of the corresponding ISAR training image.

9. A neural network-based 3D reconstruction device for ISAR image sequences, used to implement the neural network-based 3D reconstruction method for ISAR image sequences as described in any one of claims 1 to 8, characterized in that, The device includes: The training data acquisition module is used to acquire ISAR training data, which includes multiple frames of ISAR training images for a target arranged in chronological order, as well as imaging plane vectors and calibration values ​​corresponding to each ISAR training image. The six-dimensional function construction module is used to construct a six-dimensional function of the target space in the corresponding ISAR training image based on the imaging plane vector and calibration value of each ISAR training image. The six-dimensional function is the spatial position coordinates and observation direction. The voxel density and three-dimensional complex scattering coefficient acquisition module is used to input the six-dimensional function corresponding to each ISAR training image into the MLP neural network, and output the voxel density and three-dimensional complex scattering coefficient corresponding to the input six-dimensional function. The loss function construction module is used to reconstruct an ISAR rendered image based on the three-dimensional complex scattering coefficients, and to construct a loss function based on the ISAR rendered image and the corresponding ISAR training image. The MLP neural network training module trains the MLP neural network according to the loss function to obtain a trained MLP neural network. The six-dimensional function construction module for the target to be reconstructed in three dimensions is used to obtain the ISAR image sequence of the target to be reconstructed in three dimensions, as well as the imaging plane vector and calibration value corresponding to each ISAR image in the ISAR image sequence, and to construct the corresponding six-dimensional function based on the imaging plane vector and calibration value. The 3D reconstruction module is used to input the six-dimensional function corresponding to each ISAR image into the trained MLP neural network, output the corresponding voxel density and 3D complex scattering coefficient, and convert the voxel density into a 3D surface model of the target to be reconstructed according to the Marching Cube algorithm to achieve 3D reconstruction.

Images