Space target three-dimensional reconstruction method and system based on ISAR-visible light double-branch fusion neural radiation field

Abstract

The invention discloses a space target three-dimensional reconstruction method and system based on an ISAR-visible light double-branch fusion neural radiation field. The method comprises the steps of generating a visible light image and an ISAR image of a space target, wherein the visible light image and the ISAR image have the same observation view angle; inputting the ISAR image and the normal vector of the imaging plane of the ISAR image into the double-branch network structure to obtain the volume density, the amplitude and the phase of the corresponding sampling point; inputting the visible light image and the line-of-sight direction into the double-branch network structure to obtain the volume density and the color value of the corresponding sampling point; rendering the output information of the double-branch network structure to obtain a rendered image; calculating a joint loss function between the rendered image and the input image; and obtaining the volume density of the space target by using the trained double-branch network structure, and realizing three-dimensional reconstruction of the space target through a triangular mesh algorithm. According to the method, the precision of three-dimensional reconstruction of the space target is improved, the effect of multi-modal data fusion is improved, and the multi-angle observation requirement is met.

Description

Technical Field

[0001] The present invention relates to the technical field of high-orbit space target situation awareness, and in particular to a method and system for three-dimensional reconstruction of space targets based on ISAR-visible light dual-branch fusion neural radiation field. Background Art

[0002] Three-dimensional reconstruction of space targets is a core task in the field of space situational awareness. Compared with traditional two-dimensional images, three-dimensional models can provide more detailed target information and accurate component measurement parameters, thereby achieving a comprehensive understanding of the target structure. Therefore, it is of great application significance to study and develop methods that can reconstruct the three-dimensional structure of space targets and generate multi-view two-dimensional images.

[0003] At present, the three-dimensional reconstruction of space targets mainly relies on optical imaging systems. However, with the development of radar technology, ISAR (Inverse Synthetic Aperture Radar) plays an increasingly important role in the three-dimensional reconstruction of space targets. Compared with optical imaging, radar imaging has the advantage of not being restricted by lighting and meteorological conditions. The fusion of visible light images and radar images has become an effective way to improve the accuracy and robustness of three-dimensional reconstruction. However, there are significant differences between optical cameras and radars in imaging mechanisms, resolutions, and data formats, which poses a great challenge to the data fusion of the two. In order to resolve these differences and achieve effective data fusion, it is urgent to adopt advanced multi-source data fusion technology.

[0004] As a 3D reconstruction framework based on differentiable rendering, neural radiance field has achieved remarkable results in the 3D reconstruction of optical images and has shown strong capabilities in the reconstruction of other modal data such as radar. However, in the field of 3D reconstruction of space targets, research on multimodal data fusion based on ISAR and visible light images is still in its infancy. Therefore, it is of great significance to carry out research on the application of multimodal fusion of ISAR and visible light images based on neural radiance field in 3D reconstruction of space targets. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for three-dimensional reconstruction of space targets based on ISAR-visible light dual-branch fusion neural radiation field, aiming to overcome the difference problem existing in the fusion of visible light and radar images in the prior art by fusing ISAR images with optical images, and further improve the accuracy and effect of three-dimensional reconstruction of space targets.

[0006] The present invention adopts the following technical solution: a method for three-dimensional reconstruction of space targets based on ISAR-visible light dual-branch fusion neural radiation field, comprising the following steps:

[0007] S1. Generate visible light images and ISAR images of space targets with the same observation angle, and record the sight direction and the normal vector of the ISAR imaging plane.

[0008] S2. Construct a dual-branch network structure, which includes an ISAR branch and a visible light branch. The ISAR image and the normal vector of its imaging plane are input into the ISAR branch, and sampling is performed along the normal vector direction to obtain the volume density, amplitude and phase of the corresponding sampling point; the visible light image and the line of sight direction are input into the visible light branch, and sampling is performed along the line of sight direction to obtain the volume density and color value of the corresponding sampling point.

[0009] S3. Based on the information obtained in step S2, the ISAR branch and the visible light branch are rendered along their respective sampling directions to obtain an ISAR rendered image and a visible light rendered image.

[0010] S4. Calculate the loss functions between the ISAR rendered image and the visible light rendered image and the visible light image and the ISAR image in step S1 respectively, sum the loss functions to obtain a joint loss function, perform back propagation on the dual-branch network structure, optimize the parameters of the network structure, and obtain a trained dual-branch network structure.

[0011] S5. The volume density of the space target is solved by using the trained dual-branch network structure. The ISAR branch and the visible light branch respectively use the triangular mesh algorithm to map the volume density of each voxel into a surface mesh structure in three-dimensional space to achieve three-dimensional reconstruction of the space target. At the same time, the line of sight direction of any perspective is input into the trained dual-branch network structure to obtain two-dimensional images of the three-dimensional structure of the space target at different perspectives.

[0012] Furthermore, in step S1, an observation viewing angle is randomly generated, and the normal vector of the ISAR imaging plane is calculated; a visible light image under the corresponding viewing angle is generated by Blender software, and an ISAR image under the corresponding viewing angle, including amplitude and phase, is generated by MATLAB based on a range-Doppler algorithm.

[0013] Furthermore, in step S2, the sampling includes the following contents:

[0014] S201, ISAR branch and visible light branch each include two multi-layer perceptrons.

[0015] ISAR image is represented as O rad , the normal vector of the imaging plane is represented by n, n = (n x ,n y ,n z ), where n x Represents the coordinate value of the normal vector on the x-axis, n y Represents the coordinate value of the normal vector on the y axis, nz Represents the coordinate value of the normal vector on the z-axis.

[0016] The visible light image is represented by O img , the sight direction is denoted as d o , d o =(θ o ,φ o ), where θ o represents the pitch angle, φ o Indicates the azimuth.

[0017] In S202 and ISAR branches, the target point x r0 As the starting point, along n in the interval [t l ,t u ] N points are sampled equidistantly within the sampled area, and the three-dimensional coordinates of the sampling points are expressed as:

[0018]

[0019] Among them, x r (k) represents the three-dimensional coordinates of the kth sampling point in the ISAR branch, x r (k)=(x rk ,y rk ,z rk ), x rk represents the x-axis coordinate of the k-th sampling point in the ISAR branch, and y rk represents the y-axis coordinate of the kth sampling point in the ISAR branch, z rk represents the z-axis coordinate of the kth sampling point in the ISAR branch, t l represents the initial distance of the sampling points in the ISAR branch, t u represents the end distance of the sampling point in the ISAR branch, x r0 Indicates that the coordinates in the ISAR image are (p r ,p d )’s 2D pixel point corresponds to the 3D space coordinates, p r represents the range coordinate in the ISAR image, p d represents the azimuth coordinates in the ISAR image, H r Indicates the height of the ISAR image, W r represents the width of the ISAR image, Δr represents the range resolution, represents the unit vector in the range direction, Δf represents the azimuth resolution, Represents the azimuth unit vector.

[0020] In the visible light branch, the three-dimensional coordinates of the sampling points are expressed as:

[0021] x o (t) = o + t (xo0 -o)

[0022] Among them, x o (t) represents the three-dimensional coordinates of the tth sampling point in the visible light branch, x o (t) = (x ot ,y ot ,z ot ), x ot represents the x-axis coordinate of the tth sampling point in the visible light branch, y ot represents the y-axis coordinate of the tth sampling point in the visible light branch, z ot represents the z-axis coordinate of the tth sampling point in the visible light branch, o represents the center point of the camera, that is, the starting position of the light, and x o0 Indicates that the coordinates in the visible light image are (p u ,p v )’s 2D pixel point corresponds to the 3D space coordinates, p u Represents the distance coordinate in the visible light image, p v represents the azimuth coordinates in the visible light image, W o represents the width of the visible light image, f represents the focal length, H o Represents the height of the visible light image, and R represents the rotation matrix from pixel coordinates to three-dimensional space.

[0023] S203, input the three-dimensional coordinates of the sampling point and the corresponding normal vector into the ISAR branch to obtain the volume density, amplitude and phase. The specific expression is:

[0024] F ISAR (x r ,n)={s=(rp,ip),σ ISAR}

[0025] Among them, F ISAR (·) represents the multi-layer perceptron of the ISAR branch, x r represents the three-dimensional coordinates of the sampling points in the ISAR branch, σ ISAR represents the volume density of the sampling points in the ISAR branch, s represents the amplitude and phase of the sampling points in the ISAR branch, rp represents the real part, and ip represents the imaginary part.

[0026] The three-dimensional coordinates of the sampling point and the corresponding sight direction are input into the visible light branch to obtain the volume density and color value. The specific expression is:

[0027] F opt (x o ,d o )={c=(r,g,b),σ opt}

[0028] Among them, Fopt (·) represents the multilayer perceptron of the visible light branch, x o represents the three-dimensional coordinates of the sampling point in the visible light branch, c represents the color value of the sampling point in the visible light branch, r represents the red channel value, g represents the green channel value, b represents the blue channel value, σ opt Represents the volume density of sample points in the visible branch.

[0029] Furthermore, the multi-layer perceptron of the ISAR branch is divided into two parts. The first part includes 8 fully connected layers, each of which includes 256 channels and uses the ReLU function as the activation function; the second part includes a fully connected layer with 128 channels and uses the Tanh activation function.

[0030] After the sampling point coordinates are position-encoded and passed through the first part, the volume density σ is obtained ISAR And a 256-dimensional feature vector, which is concatenated with the normal direction of the imaging plane and passed through the second part to obtain the amplitude and phase s.

[0031] The multilayer perceptron of the visible light branch is divided into two parts. The first part includes 8 fully connected layers, each of which includes 256 channels and uses the ReLU function as the activation function; the second part includes a fully connected layer with 128 channels and uses the ReLU function as the activation function.

[0032] After the sampling point coordinates are position-encoded and passed through the first part, the volume density σ is obtained opt And a 256-dimensional feature vector, which is concatenated with the sight direction and passed through the second part to obtain the color information c.

[0033] Furthermore, in step S3, the rendered image is obtained including the following contents:

[0034] ISAR Rendering The calculation formula is:

[0035]

[0036] in, Represents the range coordinate of the ISAR rendered image; represents the azimuth coordinates of the ISAR rendered image; T r (k) represents the cumulative transmittance of the kth sampling point along n in the ISAR branch, σ ISAR (k) represents the volume density of the kth sampling point in the ISAR branch; s(x r (k),n) represents x r (k) The corresponding amplitude and phase.

[0037] Visible light rendering The calculation formula is:

[0038]

[0039] Among them, T o (t) represents the cumulative transmittance from the camera to the tth sampling point in the visible light branch along the line of sight, Represents the horizontal coordinate of the visible light rendered image; Represents the vertical coordinate of the visible light rendering image; σ opt (t) represents the volume density of the tth sampling point in the visible light branch, c(x o (t),d o ) represents x o (t) The corresponding color value.

[0040] Furthermore, in step S4, the trained dual-branch network structure includes the following contents:

[0041] The loss function L in the ISAR branch ISAR The calculation formula is:

[0042]

[0043] in, Indicates the number of rays processed in each batch, r ISAR represents the ray in the ISAR branch, abs represents the absolute value, Represents r ISAR ISAR rendering image corresponding to the direction, I ISAR (r ISAR ) represents the corresponding ISAR image.

[0044] Loss function L in the visible light branch opt The calculation formula is:

[0045]

[0046] Among them, r opt represents rays in the visible light branch, Represents r opt The corresponding visible light rendering image, I opt (r opt ) represents the corresponding visible light image.

[0047] The calculation formula of the joint loss function Loss is:

[0048] Loss=0.5*L ISAR +0.5*L opt ;

[0049] The Adam optimizer is used to optimize the parameters of the network structure.

[0050] Furthermore, the present invention also proposes a space target three-dimensional reconstruction system based on ISAR-visible light dual-branch fusion neural radiation field, comprising:

[0051] The information generation module is used to generate visible light images and ISAR images of space targets with the same observation angle, and record the sight direction and the normal vector of the ISAR imaging plane.

[0052] The sampling point information acquisition module is used to construct a dual-branch network structure, which includes an ISAR branch and a visible light branch. The ISAR image and the normal vector of its imaging plane are input into the ISAR branch, and sampling is performed along the normal vector direction to obtain the volume density, amplitude and phase of the corresponding sampling point; the visible light image and the line of sight direction are input into the visible light branch, and sampling is performed along the line of sight direction to obtain the volume density and color value of the corresponding sampling point.

[0053] The rendering image acquisition module is used to render the ISAR branch and the visible light branch along their respective sampling directions based on the information obtained in the sampling point information acquisition module to obtain the ISAR rendering image and the visible light rendering image.

[0054] The network structure training module is used to calculate the loss functions between the ISAR rendered image and the visible light rendered image and the visible light image and the SAR image in the information generation module respectively, and sum the loss functions to obtain the joint loss function, perform backpropagation on the two-branch network structure, optimize the parameters of the network structure, and obtain the trained two-branch network structure.

[0055] The space target three-dimensional reconstruction module is used to solve the volume density of the space target using the trained dual-branch network structure. The ISAR branch and the visible light branch use triangular mesh algorithms to map the volume density of each voxel into a surface mesh structure in three-dimensional space to achieve three-dimensional reconstruction of the space target. At the same time, the line of sight direction of any perspective is input into the trained dual-branch network structure to obtain two-dimensional images of the three-dimensional structure of the space target at different perspectives.

[0056] Furthermore, the present invention also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method for three-dimensional reconstruction of spatial targets based on ISAR-visible light dual-branch fusion neural radiation field are implemented.

[0057] Furthermore, the present invention also proposes a computer-readable storage medium, which stores a computer program. When the computer program is run by a processor, it executes the three-dimensional reconstruction method of spatial targets based on ISAR-visible light dual-branch fusion neural radiation field.

[0058] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0059] 1. By fusing ISAR images and visible light images, the present invention can effectively combine the advantages of the two imaging modes, overcome the limitations of a single technology and a single observation angle, and thus improve the accuracy of three-dimensional reconstruction of space targets.

[0060] 2. The neural radiation framework adopted in the present invention can effectively fuse data from different sensors, solve the differences between visible light imaging and radar imaging in terms of resolution, data format and imaging mechanism, and improve the effect of multimodal data fusion.

[0061] 3. The trained dual-branch neural network of the present invention can not only generate ISAR and visible light images from any perspective, but also reconstruct the three-dimensional structure of the target. It has strong multi-perspective image generation capabilities and meets multi-angle observation needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is an overall implementation flow chart of the present invention.

[0063] Figure 2 It is a schematic diagram of the ISAR and visible light imaging principles of the present invention.

[0064] Figure 3 It is a schematic diagram of the ISAR and visible light image sampling strategy of the present invention.

[0065] Figure 4 This is the ISAR branch three-dimensional reconstruction result diagram of the present invention.

[0066] Figure 5 This is the result diagram of the three-dimensional reconstruction of the visible light branch of the present invention.

[0067] Figure 6 This is a rendering result diagram of the new viewing angle test of the ISAR branch of the present invention.

[0068] Figure 7 This is a rendering result diagram of the new viewing angle test of the visible light branch of the present invention. DETAILED DESCRIPTION

[0069] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0070] To achieve the above objectives, the present invention proposes a method for three-dimensional reconstruction of space targets based on ISAR-visible light dual-branch fusion neural radiation field. Figure 1 As shown, the specific steps are as follows:

[0071] S1. Generate multiple sets of visible light images and ISAR image pairs of space targets with the same observation angle, and record the sight direction and the normal vector of the ISAR imaging plane. The specific contents are:

[0072] The observation angle of view is randomly generated, and the normal vector of the ISAR imaging plane is calculated. The visible light image under the corresponding angle of view is generated by Blender software, and the ISAR image under the corresponding angle of view, including amplitude and phase, is generated by MATLAB based on the range-Doppler algorithm.

[0073] Figure 2 The schematic diagram of visible light imaging and ISAR imaging when the observation angle is the same is shown. The ISAR imaging plane is along the line of sight, and the visible light imaging plane is perpendicular to the line of sight. The ISAR imaging plane and the visible light imaging plane are perpendicular to each other, and can invert the three-dimensional structure of the target.

[0074] In this embodiment, 50 pairs of visible light images and ISAR images with the same observation angle are generated for training.

[0075] S2. Construct a dual-branch network structure, which includes an ISAR branch and a visible light branch. Input the ISAR image and the normal vector of its imaging plane into the ISAR branch, sample along the normal vector direction, and obtain the volume density, amplitude, and phase of the corresponding sampling point; input the visible light image and the line of sight direction into the visible light branch, sample along the line of sight direction, and obtain the volume density and color value of the corresponding sampling point. The specific contents are:

[0076] S201, ISAR branch and visible light branch each include two multi-layer perceptrons;

[0077] Among them, the multi-layer perceptron of the ISAR branch is divided into two parts. The first part includes 8 fully connected layers, each of which includes 256 channels and uses the ReLU function as the activation function; the second part includes a fully connected layer with 128 channels and uses the Tanh activation function.

[0078] The sampling point coordinates are position-encoded and passed through the first part to obtain the volume density and a 256-dimensional feature vector. The feature vector is spliced ​​with the normal direction of the imaging plane and passed through the second part to obtain the amplitude and phase.

[0079] The multilayer perceptron of the visible light branch is divided into two parts. The first part includes 8 fully connected layers, each of which includes 256 channels and uses the ReLU function as the activation function; the second part includes a fully connected layer with 128 channels and uses the ReLU function as the activation function.

[0080] The sampling point coordinates are position-encoded and passed through the first part to obtain volume density and a 256-dimensional feature vector. The feature vector is concatenated with the line of sight and passed through the second part to obtain color information.

[0081] ISAR image is represented as O rad , the normal vector of the imaging plane is represented by n, n = (n x ,n y ,n z ), where n x Represents the coordinate value of the normal vector on the x-axis, n y Represents the coordinate value of the normal vector on the y axis, n z Represents the coordinate value of the normal vector on the z-axis.

[0082] The visible light image is represented by O img , the sight direction is denoted as d o , d o =(θ o ,φ o ), where θ o represents the pitch angle, φ o Indicates the azimuth.

[0083] S202, such as Figure 3 As shown, in the ISAR branch, the target point x r0 As the starting point, along n in the interval [t l ,t u ] N points are sampled equidistantly within the sampled area, and the three-dimensional coordinates of the sampling points are expressed as:

[0084]

[0085] Among them, x r (k) represents the three-dimensional coordinates of the kth sampling point in the ISAR branch, x r (k)=(x rk ,y rk ,z rk ), x rk represents the x-axis coordinate of the k-th sampling point in the ISAR branch, and y rk represents the y-axis coordinate of the kth sampling point in the ISAR branch, z rk represents the z-axis coordinate of the kth sampling point in the ISAR branch, t l represents the initial distance of the sampling points in the ISAR branch, t urepresents the end distance of the sampling point in the ISAR branch, x r0 Indicates that the coordinates in the ISAR image are (p r ,p d )’s 2D pixel point corresponds to the 3D space coordinates, p r represents the range coordinate in the ISAR image, p d represents the azimuth coordinates in the ISAR image, H r Indicates the height of the ISAR image, W r represents the width of the ISAR image, Δr represents the range resolution, represents the unit vector in the range direction, Δf represents the azimuth resolution, Represents the azimuth unit vector.

[0086] In the visible light branch, the three-dimensional coordinates of the sampling points are expressed as:

[0087] x o (t) = o + t (x o0 -o)

[0088] Among them, x o (t) represents the three-dimensional coordinates of the tth sampling point in the visible light branch, x o (t) = (x ot ,y ot ,z ot ), x ot represents the x-axis coordinate of the tth sampling point in the visible light branch, y ot represents the y-axis coordinate of the tth sampling point in the visible light branch, z ot represents the z-axis coordinate of the tth sampling point in the visible light branch, o represents the center point of the camera, that is, the starting position of the light, and x o0 Indicates that the coordinates in the visible light image are (p u ,p v )’s 2D pixel point corresponds to the 3D space coordinates, p u Represents the distance coordinate in the visible light image, p v represents the azimuth coordinates in the visible light image, W o represents the width of the visible light image, f represents the focal length, H o Represents the height of the visible light image, and R represents the rotation matrix from pixel coordinates to three-dimensional space.

[0089] S203, input the three-dimensional coordinates of the sampling point and the corresponding normal vector into the ISAR branch to obtain the volume density, amplitude and phase. The specific expression is:

[0090] F ISAR (x r,n)={s=(rp,ip),σ ISAR}

[0091] Among them, F ISAR (·) represents the multi-layer perceptron of the ISAR branch, x r represents the three-dimensional coordinates of the sampling points in the ISAR branch, σ ISAR represents the volume density of the sampling points in the ISAR branch, s represents the amplitude and phase of the sampling points in the ISAR branch, rp represents the real part, and ip represents the imaginary part.

[0092] The three-dimensional coordinates of the sampling point and the corresponding sight direction are input into the visible light branch to obtain the volume density and color value. The specific expression is:

[0093] F opt (x o ,d o )={c=(r,g,b),σ opt}

[0094] Among them, F opt (·) represents the multilayer perceptron of the visible light branch, x o represents the three-dimensional coordinates of the sampling point in the visible light branch, c represents the color value of the sampling point in the visible light branch, r represents the red channel value, g represents the green channel value, b represents the blue channel value, σ opt Represents the volume density of sample points in the visible branch.

[0095] S3, based on the information obtained in step S2, the ISAR branch and the visible light branch are rendered along their respective sampling directions to obtain an ISAR rendered image and a visible light rendered image. The specific contents are:

[0096] ISAR Rendering The calculation formula is:

[0097]

[0098] in, Represents the range coordinate of the ISAR rendered image; represents the azimuth coordinates of the ISAR rendered image; T r (k) represents the cumulative transmittance of the kth sampling point along n in the ISAR branch, σ ISAR (k) represents the volume density of the kth sampling point in the ISAR branch; s(x r (k),n) represents x r (k) The corresponding amplitude and phase.

[0099] Visible light rendering The calculation formula is:

[0100]

[0101] Among them, T o (t) represents the cumulative transmittance from the camera to the tth sampling point in the visible light branch along the line of sight, Represents the horizontal coordinate of the visible light rendered image; Represents the vertical coordinate of the visible light rendering image; σ opt (t) represents the volume density of the tth sampling point in the visible light branch, c(x o (t),d o ) represents x o (t) The corresponding color value.

[0102] S4, respectively calculating the loss function between the ISAR rendered image and the visible light rendered image and the visible light image and the ISAR image in step S1, and summing the loss functions to obtain a joint loss function, performing back propagation on the dual-branch network structure, optimizing the parameters of the network structure, and obtaining a trained dual-branch network structure. The specific contents are:

[0103] The loss function L in the ISAR branch ISAR The calculation formula is:

[0104]

[0105] in, Indicates the number of rays processed in each batch, r ISAR represents the ray in the ISAR branch, abs represents the absolute value, Represents r ISAR ISAR rendering image corresponding to the direction, I ISAR (r ISAR ) represents the corresponding ISAR image.

[0106] Loss function L in the visible light branch opt The calculation formula is:

[0107]

[0108] Among them, r opt represents rays in the visible light branch, Represents r opt The corresponding visible light rendering image, I opt (r opt ) represents the corresponding visible light image.

[0109] The calculation formula of the joint loss function Loss is:

[0110] Loss=0.5*L ISAR +0.5*L opt .

[0111] The Adam optimizer is used to optimize the parameters of the network structure, and the default parameter β is used. 1 = 0.9 and β 2 = 0.99, and the learning rate is set to 5 × 10 -5 .

[0112] S5. The volume density of the space target is solved by using the trained dual-branch network structure. The ISAR branch and the visible light branch respectively use the triangular mesh algorithm to map the volume density of each voxel into a surface mesh structure in three-dimensional space to achieve three-dimensional reconstruction of the space target. At the same time, the line of sight direction of any perspective is input into the trained dual-branch network structure to obtain the ISAR image and visible light image under the corresponding perspective. By changing the line of sight direction, the three-dimensional structure of the target can be rendered and observed from different angles.

[0113] The 3D reconstruction results and three-view images in the ISAR branch are as follows Figure 4 As shown, Figure 4 (a) is a three-dimensional image of the reconstruction result. Figure 4 (b) is the main view of the reconstruction result. Figure 4 (c) is a top view of the reconstruction result. Figure 4 (d) is a side view of the reconstruction result.

[0114] The 3D reconstruction results and three-view images of the visible light branch are as follows: Figure 5 As shown, Figure 5 (a) is a three-dimensional image of the reconstruction result. Figure 5 (b) is the main view of the reconstruction result. Figure 5 (c) is a top view of the reconstruction result. Figure 5 (d) is a side view of the reconstruction result.

[0115] from Figure 4 , 5 It can be seen that both the ISAR branch and the visible light branch completely reconstruct the target structure, verifying the effectiveness of the method proposed in the present invention.

[0116] Four different viewing angles were selected for testing. The test results of the ISAR branch and the visible light branch are as follows: Figure 6 , Figure 7 shown. Figure 6 Each row in represents a viewing angle, the first column shows the true value of the ISAR image, and the second and third columns show the ISAR rendered images without and after fusion of visible light information, respectively. Figure 6The rendering results show that the reconstruction quality of ISAR images has been significantly improved, especially in terms of enhancing the clarity of the sailboard details. The ISAR rendered image that integrates visible light information is closer to the true value, and the sailboard outline and texture in the image are clearer, while the rendered image without integrating visible light information is relatively blurred.

[0117] Figure 7 Each row in represents a viewing angle, the first column shows the true value of the visible light image, and the second and third columns show the visible light rendering image without ISAR information fusion and after ISAR information fusion, respectively. Figure 7 From the rendering results, we can see that for visible light images, the improvement effect is moderate and not as great as that of the ISAR branch, but the reconstructed texture details of the sailboard are clearer, and there are fewer shadows at the connection between the antenna and the main body, which is closer to the true value.

[0118] The embodiment of the present invention also proposes a space target three-dimensional reconstruction system based on ISAR-visible light dual-branch fusion neural radiation field, including an information generation module, a sampling point information acquisition module, a rendering image acquisition module, a network structure training module, a space target three-dimensional reconstruction module and a computer program that can be run on a processor. It should be noted that each module in the above system corresponds to the specific steps of the method provided in the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present invention.

[0119] The embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. It should be noted that when the processor executes the computer program, the specific steps of the method provided in the embodiment of the present invention are corresponding to the specific steps of the method provided in the embodiment of the present invention, and the processor has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided in the embodiment of the present invention.

[0120] The embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. It should be noted that when the computer program is executed by the processor, it corresponds to the specific steps of the method provided in the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided in the embodiment of the present invention.

[0121] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0122] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0123] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0124] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0125] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for three-dimensional reconstruction of space targets based on ISAR-visible light dual-branch fusion neural radiation field, characterized in that: include: S1, generate visible light images and ISAR images of space targets with the same observation angle, and record the sight direction and the normal vector of the ISAR imaging plane; S2. Construct a dual-branch network structure, which includes an ISAR branch and a visible light branch. Input the ISAR image and the normal vector of its imaging plane into the ISAR branch, perform sampling along the normal vector direction, and obtain the volume density, amplitude, and phase of the corresponding sampling point; input the visible light image and the line of sight direction into the visible light branch, perform sampling along the line of sight direction, and obtain the volume density and color value of the corresponding sampling point; S3, based on the information obtained in step S2, the ISAR branch and the visible light branch are rendered along their respective sampling directions to obtain an ISAR rendered image and a visible light rendered image; S4, respectively calculating the loss functions between the ISAR rendered image and the visible light rendered image and the visible light image and the ISAR image in step S1, and summing the loss functions to obtain a joint loss function, performing back propagation on the dual-branch network structure, optimizing the parameters of the network structure, and obtaining a trained dual-branch network structure; S5. The volume density of the space target is solved by using the trained dual-branch network structure. The ISAR branch and the visible light branch respectively use the triangular mesh algorithm to map the volume density of each voxel into a surface mesh structure in three-dimensional space to achieve three-dimensional reconstruction of the space target. At the same time, the line of sight direction of any perspective is input into the trained dual-branch network structure to obtain two-dimensional images of the three-dimensional structure of the space target at different perspectives.

2. The method for three-dimensional reconstruction of space targets based on ISAR-visible light dual-branch fusion neural radiation field according to claim 1 is characterized in that: In step S1, the observation angle of view is randomly generated, and the normal vector of the ISAR imaging plane is calculated; the visible light image at the corresponding angle of view is generated by Blender software, and the ISAR image at the corresponding angle of view, including amplitude and phase, is generated by MATLAB based on the range-Doppler algorithm.

3. The method for three-dimensional reconstruction of space targets based on ISAR-visible light dual-branch fusion neural radiation field according to claim 1 is characterized in that: In step S2, sampling includes the following: S201, ISAR branch and visible light branch each include two multi-layer perceptrons; ISAR image is represented as O rad , the normal vector of the imaging plane is represented by n, n = (n x ,n y ,n z ), where n x Represents the coordinate value of the normal vector on the x-axis, n y Represents the coordinate value of the normal vector on the y axis, n z Represents the coordinate value of the normal vector z-axis; The visible light image is represented by O img , the sight direction is denoted as d o , d o =(θ o ,φ o ), where θ o represents the pitch angle, φ o Indicates azimuth; In S202 and ISAR branches, the target point x r0 As the starting point, along n in the interval [t l ,t u ] N points are sampled equidistantly within the sampled area, and the three-dimensional coordinates of the sampling points are expressed as: Among them, x r (k) represents the three-dimensional coordinates of the kth sampling point in the ISAR branch, x r (k)=(x rk ,y rk ,z rk ), x rk represents the x-axis coordinate of the k-th sampling point in the ISAR branch, and y rk represents the y-axis coordinate of the kth sampling point in the ISAR branch, z rk represents the z-axis coordinate of the kth sampling point in the ISAR branch, t l represents the initial distance of the sampling points in the ISAR branch, t u represents the end distance of the sampling point in the ISAR branch, x r0 Indicates that the coordinates in the ISAR image are (p r ,p d )’s 2D pixel point corresponds to the 3D space coordinates, p r represents the range coordinate in the ISAR image, p d represents the azimuth coordinates in the ISAR image, H r Indicates the height of the ISAR image, W r represents the width of the ISAR image, Δr represents the range resolution, represents the unit vector in the range direction, Δf represents the azimuth resolution, represents the azimuthal unit vector; In the visible light branch, the three-dimensional coordinates of the sampling points are expressed as: x o (t)=o+t(x o0 -o) Among them, x o (t) represents the three-dimensional coordinates of the tth sampling point in the visible light branch, x o (t) = (x ot ,y ot ,z ot ), x ot represents the x-axis coordinate of the tth sampling point in the visible light branch, y ot represents the y-axis coordinate of the tth sampling point in the visible light branch, z ot represents the z-axis coordinate of the tth sampling point in the visible light branch, o represents the center point of the camera, that is, the starting position of the light, and x o0 Indicates that the coordinates in the visible light image are (p u ,p v )’s 2D pixel point corresponds to the 3D space coordinates, p u Represents the distance coordinate in the visible light image, p v represents the azimuth coordinates in the visible light image, W o represents the width of the visible light image, f represents the focal length, H o Represents the height of the visible light image, and R represents the rotation matrix from pixel coordinates to three-dimensional space; S203, input the three-dimensional coordinates of the sampling point and the corresponding normal vector into the ISAR branch to obtain the volume density, amplitude and phase. The specific expression is: F ISAR (x r ,n)={s=(rp,ip),σ ISAR } Among them, F ISAR (·) represents the multi-layer perceptron of the ISAR branch, x r represents the three-dimensional coordinates of the sampling points in the ISAR branch, σ ISAR represents the volume density of the sampling points in the ISAR branch, s represents the amplitude and phase of the sampling points in the ISAR branch, rp represents the real part, and ip represents the imaginary part; The three-dimensional coordinates of the sampling point and the corresponding sight direction are input into the visible light branch to obtain the volume density and color value. The specific expression is: F opt (x o ,d o )={c=(r,g,b),σ opt } Among them, F opt (·) represents the multilayer perceptron of the visible light branch, x o represents the three-dimensional coordinates of the sampling point in the visible light branch, c represents the color value of the sampling point in the visible light branch, r represents the red channel value, g represents the green channel value, b represents the blue channel value, σ opt Represents the volume density of sample points in the visible branch.

4. The method for three-dimensional reconstruction of space targets based on ISAR-visible light dual-branch fusion neural radiation field according to claim 3 is characterized in that: The multi-layer perceptron of the ISAR branch is divided into two parts. The first part includes 8 fully connected layers, each of which includes 256 channels and uses the ReLU function as the activation function; the second part includes a fully connected layer with 128 channels and uses the Tanh activation function; After the sampling point coordinates are position-encoded and passed through the first part, the volume density σ is obtained ISAR and a 256-dimensional feature vector, which is concatenated with the normal direction of the imaging plane and passed through the second part to obtain the amplitude and phase s; The multi-layer perceptron of the visible light branch is divided into two parts. The first part includes 8 fully connected layers, each of which includes 256 channels and uses the ReLU function as the activation function; the second part includes a fully connected layer with 128 channels and uses the ReLU function as the activation function; After the sampling point coordinates are position-encoded and passed through the first part, the volume density σ is obtained opt And a 256-dimensional feature vector, which is concatenated with the sight direction and passed through the second part to obtain the color information c.

5. The method for three-dimensional reconstruction of space targets based on ISAR-visible light dual-branch fusion neural radiation field according to claim 1 is characterized in that: In step S3, the rendered image is obtained including the following contents: ISAR Rendering The calculation formula is: in, Represents the range coordinate of the ISAR rendered image; Represents the azimuth coordinates of the ISAR rendered image; t l represents the initial distance of the sampling points in the ISAR branch; t u represents the end distance of the sampling point in the ISAR branch; n represents the normal vector of the imaging plane; T r (k) represents the cumulative transmittance of the kth sampling point along n in the ISAR branch, σ ISAR (k) represents the volume density of the kth sampling point in the ISAR branch; x r (k) represents the three-dimensional coordinates of the kth sampling point in the ISAR branch; s(x r (k),n) represents x r (k) the corresponding amplitude and phase; Visible light rendering The calculation formula is: Among them, T o (t) represents the cumulative transmittance from the camera to the tth sampling point in the visible light branch along the line of sight, The horizontal coordinate representing the visible light rendered image; Represents the vertical coordinate of the visible light rendering image; σ opt (t) represents the volume density of the tth sampling point in the visible light branch, x o (t) represents the three-dimensional coordinates of the tth sampling point in the visible light branch, d o represents the sight direction, c(x o (t),d o ) represents x o (t) The corresponding color value.

6. The method for three-dimensional reconstruction of space targets based on ISAR-visible light dual-branch fusion neural radiation field according to claim 1 is characterized in that: In step S4, the trained dual-branch network structure includes the following contents: The loss function L in the ISAR branch ISAR The calculation formula is: in, Indicates the number of rays processed in each batch, r ISAR represents the ray in the ISAR branch, abs represents the absolute value, Represents r ISAR ISAR rendering image corresponding to the direction, I ISAR (r ISAR ) represents the corresponding ISAR image; Loss function L in the visible light branch opt The calculation formula is: Among them, r opt represents rays in the visible light branch, Represents r opt The corresponding visible light rendering image, I opt (r opt ) represents the corresponding visible light image; The calculation formula of the joint loss function Loss is: Loss=0.5*L ISAR +0.5*L opt ; The Adam optimizer is used to optimize the parameters of the network structure.

7. A system for the method for three-dimensional reconstruction of space targets based on ISAR-visible light dual-branch fusion neural radiation field as claimed in claim 1, characterized in that: include: An information generation module is used to generate visible light images and ISAR images of space targets with the same observation angle, and record the sight direction and the normal vector of the ISAR imaging plane; The sampling point information acquisition module is used to construct a dual-branch network structure, which includes an ISAR branch and a visible light branch. The ISAR image and the normal vector of its imaging plane are input into the ISAR branch, and sampling is performed along the normal vector direction to obtain the volume density, amplitude and phase of the corresponding sampling point; the visible light image and the line of sight direction are input into the visible light branch, and sampling is performed along the line of sight direction to obtain the volume density and color value of the corresponding sampling point; A rendering image acquisition module is used to render the ISAR branch and the visible light branch along their respective sampling directions based on the information obtained in the sampling point information acquisition module to obtain an ISAR rendering image and a visible light rendering image; The network structure training module is used to calculate the loss functions between the ISAR rendered image and the visible light rendered image and the visible light image and the ISAR image in the information generation module, respectively, and sum the loss functions to obtain a joint loss function, perform back propagation on the dual-branch network structure, optimize the parameters of the network structure, and obtain a trained dual-branch network structure; The space target three-dimensional reconstruction module is used to solve the volume density of the space target using the trained dual-branch network structure. The ISAR branch and the visible light branch use triangular mesh algorithms to map the volume density of each voxel into a surface mesh structure in three-dimensional space to achieve three-dimensional reconstruction of the space target. At the same time, the line of sight direction of any perspective is input into the trained dual-branch network structure to obtain two-dimensional images of the three-dimensional structure of the space target at different perspectives.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for three-dimensional reconstruction of space targets based on ISAR-visible light dual-branch fusion neural radiation field as described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for three-dimensional reconstruction of space targets based on ISAR-visible light dual-branch fusion neural radiation field described in any one of claims 1 to 6 is executed.

Images