A space target three-dimensional reconstruction method and system based on an ISAR-visible light double-branch fusion neural radiation field

Abstract

The application discloses a kind of based on ISAR-visible light double-branch fusion neural radiation field's space target three-dimensional reconstruction method, system, the method includes generating the visible light image and ISAR image of space target with same observation visual angle;ISAR image and the normal vector of its imaging plane are input into double-branch network structure, the body density, amplitude and phase of corresponding sampling point are obtained;Visible light image and line-of-sight direction are input into double-branch network structure, the body density and color value of corresponding sampling point are obtained;The output information of double-branch network structure is rendered, and rendering image is obtained;The joint loss function between rendering image and input image is calculated;The body density of space target is obtained using the double-branch network structure trained, and the three-dimensional reconstruction of space target is realized by triangular mesh algorithm.The application improves the precision of space target three-dimensional reconstruction, improves the effect of multi-modal data fusion, meets the demand of multi-angle observation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of high orbit space target situation awareness, and particularly relates to a space target three-dimensional reconstruction method and system based on an ISAR-visible light dual-branch fusion neural radiance field. BACKGROUND

[0002] Space target three-dimensional reconstruction is a core task in the field of space situation awareness. Compared with traditional two-dimensional images, three-dimensional models can provide more detailed target information and accurate component measurement parameters, thereby achieving a comprehensive understanding of the target structure. Therefore, it is of important application significance to research and develop a method capable of reconstructing the three-dimensional structure of space targets and generating multi-view two-dimensional images.

[0003] Currently, three-dimensional reconstruction of space targets mainly relies on optical imaging systems. However, with the development of radar technology, ISAR (Inverse Synthetic Aperture Radar) plays an increasingly significant role in space target three-dimensional reconstruction. Compared with optical imaging, radar imaging has the advantage of not being limited by lighting and weather conditions. The fusion of visible light images and radar images is an effective way to improve the accuracy and robustness of three-dimensional reconstruction. However, there are significant differences between optical cameras and radars in imaging mechanisms, resolution, and data formats, which pose great challenges to the data fusion of the two. In order to address these differences and achieve effective data fusion, advanced multi-source data fusion technology is urgently needed.

[0004] Neural radiance field, as a three-dimensional reconstruction framework based on differentiable rendering, has achieved remarkable results in the three-dimensional reconstruction task of optical images and has shown strong ability in handling reconstruction tasks of other modal data such as radar. However, the research on multi-modal data fusion based on ISAR and visible light images in the field of space target three-dimensional reconstruction is still in its infancy. Therefore, it is of great significance to carry out application research on multi-modal fusion of ISAR and visible light images based on neural radiance field in space target three-dimensional reconstruction. SUMMARY

[0005] The purpose of the present application is to provide a space target three-dimensional reconstruction method and system based on ISAR-visible light dual-branch fusion neural radiance field, which aims to overcome the difference problem existing in the fusion of visible light and radar images in the prior art by fusing ISAR images and optical images, and further improve the accuracy and effect of space target three-dimensional reconstruction.

[0006] The present application adopts the following technical scheme: a space target three-dimensional reconstruction method based on ISAR-visible light dual-branch fusion neural radiance field, comprising the following steps:

[0007] S1, generate visible light images and ISAR images of the space target with the same observation view angle, and record the line-of-sight direction and the normal vector of the ISAR imaging plane.

[0008] S2, construct a double-branch network structure including an ISAR branch and a visible light branch, input the ISAR image and the normal vector of the imaging plane into the ISAR branch, sample along the normal vector direction to obtain the volume density, amplitude and phase of the corresponding sampling points, input the visible light image and the line-of-sight direction into the visible light branch, sample along the line-of-sight direction to obtain the volume density and color value of the corresponding sampling points.

[0009] S3, based on the information obtained in step S2, the ISAR branch and the visible light branch are respectively rendered along the respective sampling directions to obtain ISAR rendered images and visible light rendered images.

[0010] S4, calculate the loss function between the ISAR rendered images and the visible light rendered images and the visible light image and the ISAR image in step S1 respectively, sum the loss functions to obtain a joint loss function, perform back propagation on the double-branch network structure to optimize the parameters of the network structure, and obtain the trained double-branch network structure.

[0011] S5, use the trained double-branch network structure to solve the volume density of the space target, and use the triangular mesh algorithm to map the volume density of each voxel to the surface mesh structure in the three-dimensional space, realize the three-dimensional reconstruction of the space target; at the same time, input the line-of-sight direction of any view angle into the trained double-branch network structure to obtain the two-dimensional images of the three-dimensional structure of the space target at different view angles.

[0012] Further, in step S1, the observation view angle is randomly generated, and the normal vector of the ISAR imaging plane is calculated; the visible light image under the corresponding view angle is generated by Blender software, and the ISAR image under the corresponding view angle is generated by MATLAB based on the range-Doppler algorithm, including amplitude and phase.

[0013] Further, in step S2, the sampling includes the following contents:

[0014] S201, the ISAR branch and the visible light branch each include two multilayer perceptrons.

[0015] The ISAR image is represented as O rad , the normal vector of the imaging plane is represented as n, n=(n x ,n y ,n z ), wherein n x represents the coordinate value of the normal vector on the x-axis, n y represents the coordinate value of the normal vector on the y-axis, and nz denotes the coordinate value of the normal vector z-axis.

[0016] The visible light image is denoted as O img , and the line-of-sight direction is denoted as d o , d o = (θ o , φ o ), where θ o denotes the pitch angle, and φ o denotes the azimuth angle.

[0017] In the ISAR branch, N points are equally sampled in the interval [t l , t u ] along n with the target point x r0 as the starting point, and the three-dimensional coordinates of the sampling points are denoted as:

[0018]

[0019] where x r (k) denotes the three-dimensional coordinates of the kth sampling point in the ISAR branch, x r (k) = (x rk , y rk , z rk ), x rk denotes the x-axis coordinate of the kth sampling point in the ISAR branch, y rk denotes the y-axis coordinate of the kth sampling point in the ISAR branch, z rk denotes the z-axis coordinate of the kth sampling point in the ISAR branch, t l denotes the initial distance of the sampling points in the ISAR branch, t u denotes the end distance of the sampling points in the ISAR branch, x r0 denotes the three-dimensional space coordinates corresponding to the two-dimensional pixel point with coordinates (p r , p d ) in the ISAR image, p r denotes the range coordinate in the ISAR image, p d denotes the azimuth coordinate in the ISAR image, H r denotes the height of the ISAR image, W r denotes the width of the ISAR image, Δr denotes the range resolution, denotes the range unit vector, Δf denotes the azimuth resolution, denotes the azimuth unit vector.

[0020] In the visible light branch, the three-dimensional coordinates of the sampling points are denoted as:

[0021] x o (t) = o + t (xo0 -o)

[0022] wherein x o (t) represents the three-dimensional coordinates of the t-th sampling point in the visible light branch, x o (t) = (x ot , y ot , z ot ), x ot represents the x-axis coordinate of the t-th sampling point in the visible light branch, y ot represents the y-axis coordinate of the t-th sampling point in the visible light branch, and z ot represents the z-axis coordinate of the t-th sampling point in the visible light branch, o represents the center point of the camera, i.e., the starting position of the light ray, x o0 represents the two-dimensional pixel point in the visible light image with coordinates (p u , p v ), and p u represents the distance coordinate in the visible light image, p v represents the azimuth coordinate in the visible light image. W o represents the width of the visible light image, f represents the focal length, H o represents the height of the visible light image, and R represents the rotation matrix of the pixel coordinates to the three-dimensional space.

[0023] S203, input the three-dimensional coordinates of the sampling points and the corresponding normal vectors into the ISAR branch to obtain the volume density, amplitude, and phase, and the specific expression is:

[0024] F ISAR (x r , n) = {s = (rp, ip), σ ISAR}

[0025] wherein F ISAR (·) represents the multi-layer perception machine of the ISAR branch, x r represents the three-dimensional coordinates of the sampling points in the ISAR branch, σ ISAR represents the volume density of the sampling points in the ISAR branch, s represents the amplitude and phase of the sampling points in the ISAR branch, rp represents the real part, and ip represents the imaginary part.

[0026] Input the three-dimensional coordinates of the sampling points and the corresponding line-of-sight directions into the visible light branch to obtain the volume density and color value, and the specific expression is:

[0027] F opt (x o , d o ) = {c = (r, g, b), σ opt}

[0028] wherein Fopt (·) represents a multi-layer perception of the visible light branch, x o represents the three-dimensional coordinates of the sampling points in the visible light branch, c represents the color value of the sampling points in the visible light branch, r represents the red channel value, g represents the green channel value, b represents the blue channel value, σ opt represents the volume density of the sampling points in the visible light branch.

[0029] Further, the multi-layer perception of the ISAR branch is divided into two parts, the first part includes 8 fully connected layers, each fully connected layer includes 256 channels, and uses the ReLU function as the activation function; the second part includes a fully connected layer with 128 channels, and uses the Tanh activation function.

[0030] After position encoding of the sampling point coordinates, the first part is passed to obtain the volume density σ ISAR and a 256-dimensional feature vector, the feature vector is spliced with the imaging plane normal direction and passed through the second part to obtain the amplitude and phase s.

[0031] The multi-layer perception of the visible light branch is divided into two parts, the first part includes 8 fully connected layers, each fully connected layer includes 256 channels, and uses the ReLU function as the activation function; the second part includes a fully connected layer with 128 channels, and uses the ReLU function as the activation function.

[0032] After position encoding of the sampling point coordinates, the first part is passed to obtain the volume density σ opt and a 256-dimensional feature vector, the feature vector is spliced with the line of sight direction and passed through the second part to obtain the color information c.

[0033] Further, in step S3, obtaining the rendered image includes the following contents:

[0034] ISAR rendered image The calculation formula is:

[0035]

[0036] wherein, represents the range coordinate of the ISAR rendered image; represents the azimuth coordinate of the ISAR rendered image; T r (k) represents the cumulative transmittance of the kth sampling point in the ISAR branch along n, σ ISAR (k) represents the volume density of the kth sampling point in the ISAR branch; s(x r (k), n) represents the amplitude and phase corresponding to x r (k).

[0037] Visible light rendering image The calculation formula of L

[0038]

[0039] Wherein, T o (t) represents the cumulative transmittance from the camera to the t th sampling point in the visible light branch along the line of sight direction, represents the horizontal coordinate of the visible light rendering image; represents the vertical coordinate of the visible light rendering image; sigma opt (t) represents the body density of the t th sampling point in the visible light branch, c(x o (t), d o ) represents the color value corresponding to x o (t).

[0040] Further, in step S4, the trained double-branch network structure comprises the following contents:

[0041] The calculation formula of the loss function L ISAR in the ISAR branch is:

[0042]

[0043] Wherein, represents the number of rays processed in each batch, r ISAR represents the ray in the ISAR branch, and abs represents the absolute value, represents the ISAR rendering image corresponding to the direction of r ISAR , and I ISAR (r ISAR ) represents the corresponding ISAR image.

[0044] The calculation formula of the loss function L opt in the visible light branch is:

[0045]

[0046] Wherein, r opt represents the ray in the visible light branch, represents the visible light rendering image corresponding to r opt , and I opt (r opt ) represents the corresponding visible light image.

[0047] The calculation formula of the joint loss function Loss is:

[0048] Loss = 0.5 * L ISAR + 0.5 * L opt ;

[0049] The parameters of the network structure are optimized using an Adam optimizer.

[0050] Further, the application also proposes a spatial target three-dimensional reconstruction system based on an ISAR-visible light double-branch fusion neural radiation field, comprising:

[0051] An information generation module is configured to generate visible light images and ISAR images of the spatial target with the same observation angle, and record the line-of-sight direction and the normal vector of the ISAR imaging plane.

[0052] A sampling point information acquisition module is configured to construct a double-branch network structure, which comprises an ISAR branch and a visible light branch, input the ISAR image and the normal vector of the imaging plane into the ISAR branch, sample along the normal vector direction to obtain the volume density, amplitude and phase of the corresponding sampling points, and input the visible light image and the line-of-sight direction into the visible light branch, sample along the line-of-sight direction to obtain the volume density and color value of the corresponding sampling points.

[0053] A rendered image acquisition module is configured to render the ISAR branch and the visible light branch along their respective sampling directions based on the information obtained by the sampling point information acquisition module, to obtain ISAR rendered images and visible light rendered images.

[0054] A network structure training module is configured to calculate the loss function between the ISAR rendered images and the visible light rendered images and the visible light images and the SAR images in the information generation module respectively, sum the loss functions to obtain a joint loss function, perform back propagation on the double-branch network structure, optimize the parameters of the network structure, and obtain the trained double-branch network structure.

[0055] A spatial target three-dimensional reconstruction module is configured to solve the volume density of the spatial target by using the trained double-branch network structure, and use the triangular mesh algorithm by the ISAR branch and the visible light branch to map the volume density of each voxel to the surface mesh structure in the three-dimensional space, to realize the three-dimensional reconstruction of the spatial target; and input the line-of-sight direction of any angle into the trained double-branch network structure to obtain the two-dimensional images of the three-dimensional structure of the spatial target at different angles.

[0056] Further, the application also proposes an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the spatial target three-dimensional reconstruction method based on the ISAR-visible light double-branch fusion neural radiation field when executing the computer program.

[0057] Further, the application further provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to perform the ISAR-visible light dual-branch fusion neural radiance field based spatial target three-dimensional reconstruction method.

[0058] Compared with the prior art, the application has the following technical effects:

[0059] 1. The application can effectively combine the advantages of two imaging modes by fusing ISAR images and visible light images, overcome the limitations of single technology and single observation angle, and improve the accuracy of spatial target three-dimensional reconstruction.

[0060] 2. The neural radiance framework used in the application can effectively fuse data from different sensors, solve the differences in resolution, data format and imaging mechanism between visible light imaging and radar imaging, and improve the effect of multi-modal data fusion.

[0061] 3. The dual-branch neural network trained by the application can not only generate ISAR and visible light images from any angle, but also reconstruct the three-dimensional structure of the target, has strong multi-view image generation capability, and meets the multi-angle observation demand. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 is the overall implementation flowchart of the application.

[0063] Figure 2 is the ISAR and visible light imaging principle diagram of the application.

[0064] Figure 3 is the ISAR and visible light image sampling strategy diagram of the application.

[0065] Figure 4 is the ISAR branch three-dimensional reconstruction result diagram of the application.

[0066] Figure 5 is the visible light branch three-dimensional reconstruction result diagram of the application.

[0067] Figure 6 is the ISAR branch new angle test rendering result diagram of the application.

[0068] Figure 7 is the visible light branch new angle test rendering result diagram of the application. DETAILED DESCRIPTION

[0069] The application will be further described below in combination with the drawings. The following examples are only used to more clearly illustrate the technical solutions of the application, and cannot be used to limit the protection scope of the application.

[0070] To achieve the above object, the application provides a spatial target three-dimensional reconstruction method based on ISAR-visible light double-branch fusion neural radiation field, as shown in the figure, and the specific steps are as follows: Figure 1

[0071] S1, a plurality of groups of visible light images and ISAR images of the spatial target with the same observation angle are generated, and the line-of-sight direction and the normal vector of the ISAR imaging plane are recorded. The specific content is:

[0072] The observation angle is randomly generated, the normal vector of the ISAR imaging plane is calculated, the visible light image under the corresponding angle is generated through the Blender software, and the ISAR image under the corresponding angle is generated based on the range-Doppler algorithm and by using MATLAB, including the amplitude and the phase.

[0073] Figure 2 The principle diagram of visible light imaging and ISAR imaging when the observation angle is the same is shown, the ISAR imaging plane is along the line-of-sight direction, the visible light imaging plane is perpendicular to the line-of-sight direction, the ISAR imaging plane and the visible light imaging plane are perpendicular to each other, and the three-dimensional structure of the target can be inverted.

[0074] In this embodiment, 50 pairs of visible light images and ISAR images with the same observation angle are generated for training.

[0075] S2, a double-branch network structure is constructed, the network structure includes an ISAR branch and a visible light branch, the ISAR image and the normal vector of the imaging plane are input into the ISAR branch, sampling is performed along the normal vector direction to obtain the body density, the amplitude and the phase of the corresponding sampling points, and the visible light image and the line-of-sight direction are input into the visible light branch, sampling is performed along the line-of-sight direction to obtain the body density and the color value of the corresponding sampling points. The specific content is:

[0076] S201, the ISAR branch and the visible light branch both include two multilayer perceptrons;

[0077] The multilayer perceptron of the ISAR branch is divided into two parts, the first part includes 8 fully connected layers, each fully connected layer includes 256 channels, and uses the ReLU function as the activation function; the second part includes a fully connected layer with 128 channels, and uses the Tanh activation function.

[0078] After the position encoding of the sampling point coordinates is performed, the body density and a 256-dimensional feature vector are obtained through the first part, the feature vector is spliced with the normal direction of the imaging plane, and then the second part is passed through to obtain the amplitude and the phase.

[0079] ​The visible light branch of the multi-layer perception machine is divided into two parts, the first part includes 8 fully connected layers, each fully connected layer includes 256 channels, and uses a ReLU function as an activation function; the second part includes a fully connected layer with 128 channels, and uses a ReLU function as an activation function.

[0080] After position encoding the coordinates of the sampling points, the first part is passed to obtain the body density and a 256-dimensional feature vector, and the feature vector is spliced with the line-of-sight direction to pass through the second part to obtain the color information.

[0081] The ISAR image is represented as O rad , and the normal vector of the imaging plane is represented as n, n = (n x , n y , n z ), wherein n x represents the coordinate value of the normal vector on the x-axis, n y represents the coordinate value of the normal vector on the y-axis, and n z represents the coordinate value of the normal vector on the z-axis.

[0082] The visible light image is represented as O img , and the line-of-sight direction is represented as d o , d o = (θ o , φ o ), wherein θ o represents the pitch angle, and φ o represents the azimuth angle.

[0083] S202, as shown in Figure 3 , in the ISAR branch, starting from the target point x r0 , N points are sampled at equal intervals in the interval [t l , t u ] along n, and the three-dimensional coordinates of the sampling points are represented as:

[0084]

[0085] , wherein x r (k) represents the three-dimensional coordinates of the kth sampling point in the ISAR branch, x r (k) = (x rk , y rk , z rk ), x rk represents the x-axis coordinate of the kth sampling point in the ISAR branch, y rk represents the y-axis coordinate of the kth sampling point in the ISAR branch, z rk represents the z-axis coordinate of the kth sampling point in the ISAR branch, t l represents the initial distance of the sampling points in the ISAR branch, and t udenotes the end distance of the sampling points in the ISAR branch, x r0 denotes the three-dimensional space coordinates corresponding to the two-dimensional pixel point with coordinates (p r , p d ) in the ISAR image, p r denotes the range direction coordinate within the ISAR image, p d denotes the azimuth direction coordinate within the ISAR image, H r denotes the height of the ISAR image, W r denotes the width of the ISAR image, and Δr denotes the range resolution, denotes the range direction unit vector, and Δf denotes the azimuth resolution, denotes the azimuth direction unit vector.

[0086] In the visible light branch, the three-dimensional coordinates of the sampling points are denoted as:

[0087] x o (t) = o + t (x o0 -o)

[0088] wherein x o (t) denotes the three-dimensional coordinates of the t-th sampling point in the visible light branch, x o (t) = (x ot , y ot , z ot ), x ot denotes the x-axis coordinate of the t-th sampling point in the visible light branch, y ot denotes the y-axis coordinate of the t-th sampling point in the visible light branch, z ot denotes the z-axis coordinate of the t-th sampling point in the visible light branch, o denotes the center point of the camera, i.e., the starting position of the light, x o0 denotes the three-dimensional space coordinates corresponding to the two-dimensional pixel point with coordinates (p u , p v ) in the visible light image, p u denotes the range direction coordinate within the visible light image, p v denotes the azimuth direction coordinate within the visible light image, W o denotes the width of the visible light image, f denotes the focal length, H o denotes the height of the visible light image, and R denotes the rotation matrix of the pixel coordinates to the three-dimensional space.

[0089] In S203, the three-dimensional coordinates of the sampling points and the corresponding normal vectors are input into the ISAR branch to obtain the volume density, the amplitude and the phase, and the specific expression is:

[0090] F ISAR (x r,n)={s=(rp,ip),σ ISAR}

[0091] Among them, F ISAR (·) represents a multilayer perceptron in the ISAR branch, x r σ represents the three-dimensional coordinates of the sampling point in the ISAR branch. ISAR denoted by , where s represents the volume density of the sampling points in the ISAR branch, s represents the amplitude and phase of the sampling points in the ISAR branch, rp represents the real part, and ip represents the imaginary part.

[0092] The three-dimensional coordinates of the sampling point and the corresponding viewing direction are input into the visible light branch to obtain the volume density and color values, specifically expressed as follows:

[0093] F opt (x o ,d o )={c=(r,g,b),σ opt}

[0094] Among them, F opt (·) denotes a multilayer perceptron with visible light branching, x o σ represents the three-dimensional coordinates of the sampling point in the visible light branch, c represents the color value of the sampling point in the visible light branch, r represents the red channel value, g represents the green channel value, b represents the blue channel value, and σ represents the color value of the sampling point in the visible light branch. opt This represents the volume density of sampling points in the visible light branch.

[0095] S3. Based on the information obtained in step S2, the ISAR branch and the visible light branch are rendered along their respective sampling directions to obtain the ISAR rendered image and the visible light rendered image. The specific content is as follows:

[0096] ISAR rendered image The calculation formula is:

[0097]

[0098] in, Represents the range coordinates of the ISAR rendered image; T represents the azimuth coordinates of the ISAR rendered image; r (k) represents the cumulative transmittance along n at the k-th sampling point in the ISAR branch. σ ISAR (k) represents the volume density of the k-th sampling point in the ISAR branch; s(x r (k),n) represents x r (k) corresponds to the amplitude and phase.

[0099] Visible light rendering image The calculation formula is:

[0100]

[0101] where T o (t) represents the cumulative transmittance along the line-of-sight direction from the camera to the t-th sampling point in the visible light branch, represents the horizontal coordinate of the visible light rendered image; represents the vertical coordinate of the visible light rendered image;σ opt (t) represents the volume density of the t-th sampling point in the visible light branch, c(x o (t), d o ) represents the color value corresponding to x o (t).

[0102] S4, respectively calculate the loss functions between the ISAR rendered image and the visible light rendered image and the visible light image and the ISAR image in step S1, sum the loss functions to obtain a joint loss function, perform back propagation on the dual-branch network structure, optimize the parameters of the network structure, and obtain the trained dual-branch network structure. The specific content is:

[0103] The calculation formula of the loss function L ISAR in the ISAR branch is:

[0104]

[0105] wherein, represents the number of rays processed in each batch, r ISAR represents a ray in the ISAR branch, and abs represents taking an absolute value, represents the ISAR rendered image corresponding to the direction of r ISAR , and I ISAR (r ISAR ) represents the corresponding ISAR image.

[0106] The calculation formula of the loss function L opt in the visible light branch is:

[0107]

[0108] wherein, r opt represents a ray in the visible light branch, represents the visible light rendered image corresponding to r opt , and I opt (r opt ) represents the corresponding visible light image.

[0109] The calculation formula of the joint loss function Loss is:

[0110] Loss = 0.5 * L ISAR + 0.5 * L opt .

[0111] The parameters of the network structure are optimized using an Adam optimizer, and the default parameters β1 = 0.9 and β2 = 0.99 are adopted, and the learning rate is set to 5 x 10 -5 .

[0112] S5, using the trained dual-branch network structure to solve the body density of the space target, the ISAR branch and the visible light branch respectively use the triangular grid algorithm to map the body density of each voxel to the surface grid structure in the three-dimensional space, realizing the three-dimensional reconstruction of the space target; at the same time, the line-of-sight direction of any viewing angle is input into the trained dual-branch network structure, to obtain the ISAR image and the visible light image under the corresponding viewing angle, and the three-dimensional structure of the target is rendered and observed from different angles by changing the line-of-sight direction.

[0113] The three-dimensional reconstruction result and the three views in the ISAR branch are as shown in Figure 4 , wherein Figure 4 (a) of is the three-dimensional view of the reconstruction result, Figure 4 (b) of is the front view of the reconstruction result, Figure 4 (c) of is the top view of the reconstruction result, Figure 4 (d) of is the side view of the reconstruction result.

[0114] The three-dimensional reconstruction result and the three views in the visible light branch are as shown in Figure 5 , wherein Figure 5 (a) of is the three-dimensional view of the reconstruction result, Figure 5 (b) of is the front view of the reconstruction result, Figure 5 (c) of is the top view of the reconstruction result, Figure 5 (d) of is the side view of the reconstruction result.

[0115] As can be seen from Figure 4 , 5 , the ISAR branch and the visible light branch both complete the reconstruction of the target structure, verifying the effectiveness of the method proposed in the application.

[0116] Four different viewing angles are selected for testing, and the test results of the ISAR branch and the visible light branch are as shown in Figure 6 , Figure 7 . Figure 6 Each row in Figure 6It can be seen from the rendering results that the reconstruction quality of the ISAR image is significantly improved, especially in the clarity of enhancing the sailboard details. The ISAR rendered image fused with visible light information is closer to the true value, and the sailboard profile and texture in the image are clearer, while the rendered image without visible light information is relatively blurred.

[0117] Figure 7 Each row represents a viewing angle, the first column shows the true value of the visible light image, and the second and third columns show the visible light rendered image before and after fusing the ISAR information, respectively. From the rendering results, it can be seen that for the visible light image, the improvement effect is moderate, and the improvement degree without the ISAR branch is large, but the reconstructed texture details of the sailboard are clearer, the shadow at the connection between the antenna and the main body is less, and it is closer to the true value. Figure 7

[0118] The embodiment of the present application also proposes a spatial target three-dimensional reconstruction system based on the ISAR-visible light double-branch fused neural radiance field, which comprises an information generation module, a sampling point information acquisition module, a rendered image acquisition module, a network structure training module, a spatial target three-dimensional reconstruction module and a computer program executable on a processor. It should be noted that each module in the above system corresponds to the specific steps of the method provided by the embodiment of the present application, has the corresponding function modules and beneficial effects of the execution method. Technical details not described in detail in the present embodiment can be referred to the method provided by the embodiment of the present application.

[0119] The embodiment of the present application also proposes an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor. It should be noted that the processor executes the computer program corresponding to the specific steps of the method provided by the embodiment of the present application, has the corresponding function modules and beneficial effects of the execution method. Technical details not described in detail in the present embodiment can be referred to the method provided by the embodiment of the present application.

[0120] The embodiment of the present application also proposes a computer-readable storage medium, which stores a computer program. It should be noted that the computer program is run by the processor corresponding to the specific steps of the method provided by the embodiment of the present application, has the corresponding function modules and beneficial effects of the execution method. Technical details not described in detail in the present embodiment can be referred to the method provided by the embodiment of the present application.

[0121] ​Those skilled in the art will appreciate that embodiments of the application can be readily used as software, hardware, or a combination of software and hardware. In one embodiment, the application can be implemented in software and can be stored on and executed by a computer program product, for example, a computer program product that can be a memory device of a computer. In some embodiments, the computer program product can be a memory device such as a floppy disk, a computer disk, a hard disk, a memory stick, a CD-ROM, a memory chip, a RAM, a ROM, or an optical memory device, etc.

[0122] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 The flowchart blocks and / or diagrams can also represent a computer program in which the flowchart blocks and / or diagrams are one or more of the following: a process, a procedure, a function, a routine, a subroutine, a module, a segment, a thread of execution, a program, a routine, a application, a midlet, a servlet, a program, a program component, a process, a method, an algorithm, a logic, a program, a program of instructions, an instruction, an instruction set, an instruction stream, or a executable program. Figure 1 The flowchart blocks and / or diagrams can also represent a computer program in which the flowchart blocks and / or diagrams are one or more of the following: a process, a procedure, a function, a routine, a subroutine, a module, a segment, a thread of execution, a program, a routine, a application, a midlet, a servlet, a program, a program component, a process, a method, an algorithm, a logic, a program, a program of instructions, an instruction, an instruction set, an instruction stream, or a executable program.

[0123] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 The flowchart blocks and / or diagrams can also represent a computer program in which the flowchart blocks and / or diagrams are one or more of the following: a process, a procedure, a function, a routine, a subroutine, a module, a segment, a thread of execution, a program, a routine, a application, a midlet, a servlet, a program, a program component, a process, a method, an algorithm, a logic, a program, a program of instructions, an instruction, an instruction set, an instruction stream, or a executable program. Figure 1 The flowchart blocks and / or diagrams can also represent a computer program in which the flowchart blocks and / or diagrams are one or more of the following: a process, a procedure, a function, a routine, a subroutine, a module, a segment, a thread of execution, a program, a routine, a application, a midlet, a servlet, a program, a program component, a process, a method, an algorithm, a logic, a program, a program of instructions, an instruction, an instruction set, an instruction stream, or a executable program.

[0124] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 The flowchart blocks and / or diagrams can also represent a computer program in which the flowchart blocks and / or diagrams are one or more of the following: a process, a procedure, a function, a routine, a subroutine, a module, a segment, a thread of execution, a program, a routine, a application, a midlet, a servlet, a program, a program component, a process, a method, an algorithm, a logic, a program, a program of instructions, an instruction, an instruction set, an instruction stream, or a executable program. Figure 1 The flowchart blocks and / or diagrams can also represent a computer program in which the flowchart blocks and / or diagrams are one or more of the following: a process, a procedure, a function, a routine, a subroutine, a module, a segment, a thread of execution, a program, a routine, a application, a midlet, a servlet, a program, a program component, a process, a method, an algorithm, a logic, a program, a program of instructions, an instruction, an instruction set, an instruction stream, or a executable program.

[0125] The above only is the preferred embodiment of the present application, it should be pointed out that, for those skilled in the art, without departing from the technical principles of the present application, can make a number of improvements and variations, these improvements and variations should be considered as the protection scope of the present application.

Claims

1. A space target three-dimensional reconstruction method based on ISAR-visible light double-branch fusion neural radiation field, characterized by, The method comprises the following steps: S1, generating a visible light image and an ISAR image of a space target with the same observation view angle, and recording the line-of-sight direction and the normal vector of the ISAR imaging plane; S2, constructing a double-branch network structure, inputting the ISAR image and the normal vector of the imaging plane into the ISAR branch, sampling along the normal vector direction to obtain the body density, amplitude and phase of the corresponding sampling points, inputting the visible light image and the line-of-sight direction into the visible light branch, and sampling along the line-of-sight direction to obtain the body density and color value of the corresponding sampling points; S3, based on the information obtained in step S2, the ISAR branch and the visible light branch are rendered along the respective sampling directions to obtain ISAR rendered images and visible light rendered images; S4, the loss functions between the ISAR rendered images and the visible light rendered images and the visible light image and the ISAR image in step S1 are calculated respectively, the joint loss function is obtained by summing the loss functions, the double-branch network structure is back propagated, the parameters of the network structure are optimized, and the trained double-branch network structure is obtained; S5, the body density of the space target is solved by using the trained double-branch network structure, the ISAR branch and the visible light branch respectively use the triangular mesh algorithm to map the body density of each voxel to the surface mesh structure in the three-dimensional space, and the three-dimensional reconstruction of the space target is realized; meanwhile, the line-of-sight direction of any view angle is input into the trained double-branch network structure to obtain the two-dimensional images of the three-dimensional structure of the space target at different view angles.

2. The ISAR-visual light dual-branch fusion neural radiometric field based spatial target three-dimensional reconstruction method according to claim 1, characterized in that, In step S1, the observation view angle is randomly generated, and the normal vector of the ISAR imaging plane is calculated; the visible light image under the corresponding view angle is generated by using Blender software, and the ISAR image under the corresponding view angle is generated by using MATLAB based on the range-Doppler algorithm, including amplitude and phase.

3. The ISAR-visual light dual-branch fusion neural radiometric field based spatial target three-dimensional reconstruction method according to claim 1, characterized in that, In step S2, the sampling includes the following contents: S201, the ISAR branch and the visible light branch both include two multilayer perceptrons; ISAR image is represented as O rad , the normal vector of the imaging plane is represented as n, n = (n x , n y , n z ), wherein n x represents the coordinate value of the normal vector on the x-axis, n y represents the coordinate value of the normal vector on the y-axis, and n z represents the coordinate value of the normal vector on the z-axis; The visible light image is denoted as O img The line of sight direction is denoted as d o d o = (θ o , Φ o ), where θ o denotes the elevation angle and Φ o denotes the azimuth angle; S202、ISAR branch, with the target point x r0 as the starting point, N points are equally sampled in the interval [t l , t u ] along n, and the three-dimensional coordinates of the sampling points are represented as: wherein x r (k) represents the three-dimensional coordinates of the kth sampling point in the ISAR branch, x r (k) = (x rk , y rk , z rk ), x rk represents the x-axis coordinates of the kth sampling point in the ISAR branch, y rk represents the y-axis coordinates of the kth sampling point in the ISAR branch, and z rk represents the z-axis coordinates of the kth sampling point in the ISAR branch, t l represents the initial distance of the sampling points in the ISAR branch, t u represents the end distance of the sampling points in the ISAR branch, x r0 represents the two-dimensional pixel point corresponding to the three-dimensional space coordinates of (p r , p d ) in the ISAR image, p r represents the range direction coordinates within the ISAR image, p d represents the azimuth direction coordinates within the ISAR image, H r represents the height of the ISAR image, W r represents the width of the ISAR image, Δr represents the range resolution, represents the range direction unit vector, Δf represents the azimuth resolution, represents the azimuth direction unit vector; In the visible light branch, the three-dimensional coordinates of the sampling points are represented as: x o (t) = o + t(x o0 -o) wherein x o (t) represents the three-dimensional coordinates of the tth sampling point in the visible light branch, x o (t) = (x ot , y ot , z ot ), x ot represents the x-axis coordinates of the tth sampling point in the visible light branch, y ot represents the y-axis coordinates of the tth sampling point in the visible light branch, and z ot represents the z-axis coordinates of the tth sampling point in the visible light branch, o represents the center point of the camera, i.e., the starting position of the light ray, x o0 represents the three-dimensional space coordinates corresponding to the two-dimensional pixel point with coordinates (p u , p v ) in the visible light image, p u represents the distance direction coordinate within the visible light image, p v represents the azimuth direction coordinate within the visible light image, W o represents the width of the visible light image, f represents the focal length, H o represents the height of the visible light image, and R represents the rotation matrix of the pixel coordinates to the three-dimensional space. S203, the three-dimensional coordinates of the sampling points and the corresponding normal vector are input into the ISAR branch to obtain the body density, amplitude and phase, and the specific expression is: F ISAR (x r ,n)={s=(rp,ip),σ ISAR} where F ISAR (·) denotes the multi-layer perceptron of the ISAR branch, x r represents the three-dimensional coordinates of the sampling points in the ISAR branch, σ ISAR represents the volume density of the sampling points in the ISAR branch, s represents the amplitude and phase of the sampling points in the ISAR branch, rp represents the real part, and ip represents the imaginary part; The three-dimensional coordinates of the sampling points and the corresponding line-of-sight direction are input into the visible light branch to obtain the body density and color value, and the specific expression is: F opt (x o ,d o )={c=(r,g,b),σ opt} where F opt (·) denotes the multi-layer perceptron for the visible light branch, x o denotes the three-dimensional coordinates of the sampling points in the visible light branch, c denotes the color values of the sampling points in the visible light branch, r denotes the red channel numerical value, g denotes the green channel numerical value, b denotes the blue channel numerical value, σ opt denotes the volume density of the sampling points in the visible light branch.

4. The ISAR-visual light dual-branch fusion neural radiometric field based spatial target three-dimensional reconstruction method according to claim 3, characterized in that, The multilayer perceptron of the ISAR branch is divided into two parts, the first part includes 8 fully connected layers, each fully connected layer includes 256 channels, and uses the ReLU function as the activation function; the second part includes a fully connected layer with 128 channels, and uses the Tanh activation function; After position encoding the coordinates of the sampling points, the first part is passed to obtain the body density σ ISAR and a 256-dimensional feature vector, which is spliced with the imaging plane normal direction to pass through the second part to obtain the amplitude and phase s; The multilayer perceptron of the visible light branch is divided into two parts, the first part includes 8 fully connected layers, each fully connected layer includes 256 channels, and uses the ReLU function as the activation function; the second part includes a fully connected layer with 128 channels, and uses the ReLU function as the activation function; After position encoding the coordinates of the sampling points, the first part is passed to obtain the body density σ opt and a 256-dimensional feature vector, which is concatenated with the view direction and passed through the second part to obtain the color information c.

5. The ISAR-visual light dual-branch fusion neural radiometric field based spatial target three-dimensional reconstruction method according to claim 1, characterized in that, In step S3, the rendered images include the following contents: ISAR rendered image The formula for calculating is: wherein, denotes the range coordinate of the ISAR rendered image; denotes the azimuth coordinate of the ISAR rendered image; l denotes the initial distance of a sampling point in the ISAR branch; u denotes the end distance of a sampling point in the ISAR branch; n denotes the normal vector of the imaging plane; r (k) denotes the cumulative transmittance of the k-th sampling point in the ISAR branch along n, σ ISAR (k) denotes the volume density of the k-th sampling point in the ISAR branch; r (k) denotes the three-dimensional coordinates of the k-th sampling point in the ISAR branch; r (k), n) denotes the amplitude and phase of x r (k) corresponding to x Visible light rendered image The formula for calculating is: where T o (t) denotes the cumulative transmittance along the line of sight direction from the camera to the t-th sample point in the visible light branch, denotes the horizontal coordinate of the visible light rendered image; denotes the vertical coordinate of the visible light rendered image; σ opt (t) denotes the volume density of the t-th sample point in the visible light branch, x o (t) denotes the three-dimensional coordinate of the t-th sample point in the visible light branch, d o denotes the line of sight direction, c(x o (t), d o ) denotes the color value corresponding to x o (t).

6. The ISAR-visual light dual-branch fusion neural radiometric field based spatial target three-dimensional reconstruction method according to claim 1, characterized in that, In step S4, obtaining the trained double-branch network structure includes the following contents: The loss function L in the ISAR branch ISAR is calculated as wherein, represents the number of rays processed per batch, r ISAR represents the rays in the ISAR branch, abs represents taking the absolute value, represents the ISAR rendered image corresponding to r ISAR direction, I ISAR (r ISAR ) represents the corresponding ISAR image; Loss function L in visible light branch opt The calculation formula is: where r opt represents a ray in the visible light branch, represents a visible light rendered image, I opt corresponding to r opt (r opt ) represents the corresponding visible light image; The calculation formula of the joint loss function Loss is: Loss = 0.5 * L ISAR + 0.5 * L opt ; The parameters of the network structure are optimized using an Adam optimizer.

7. The system for applying the ISAR-optical dual-branch fusion neural radiance field based space target three-dimensional reconstruction method of claim 1, characterized in that, The method comprises the following steps: An information generation module is configured to generate visible light images and ISAR images of a space target with the same observation angle, and record the line-of-sight direction and the normal vector of the ISAR imaging plane; A sampling point information acquisition module is configured to construct a double-branch network structure, which comprises an ISAR branch and a visible light branch, the ISAR image and the normal vector of the ISAR imaging plane are input into the ISAR branch, sampling is performed along the normal vector direction to obtain the volume density, amplitude and phase of the corresponding sampling point, and the visible light image and the line-of-sight direction are input into the visible light branch, sampling is performed along the line-of-sight direction to obtain the volume density and color value of the corresponding sampling point; A rendered image acquisition module is configured to render the ISAR branch and the visible light branch along the respective sampling directions based on the information obtained by the sampling point information acquisition module, to obtain an ISAR rendered image and a visible light rendered image; A network structure training module is configured to calculate the loss function between the ISAR rendered image and the visible light rendered image and the visible light image and the ISAR image in the information generation module, sum the loss functions to obtain a joint loss function, perform back propagation on the double-branch network structure, optimize the parameters of the network structure, and obtain a trained double-branch network structure; A space target three-dimensional reconstruction module is configured to solve the volume density of the space target by using the trained double-branch network structure, the ISAR branch and the visible light branch respectively use a triangular mesh algorithm to map the volume density of each voxel to a surface mesh structure in a three-dimensional space, and three-dimensional reconstruction of the space target is realized; meanwhile, the line-of-sight direction of an arbitrary angle is input into the trained double-branch network structure to obtain a two-dimensional image of the three-dimensional structure of the space target at different angles.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the steps of the space target three-dimensional reconstruction method based on the ISAR-visible light double-branch fusion neural radiative field according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, the computer-readable storage medium being characterized by, The computer program is run by the processor to execute the space target three-dimensional reconstruction method based on the ISAR-visible light double-branch fusion neural radiative field according to any one of claims 1 to 6.

Images