Space target three-dimensional reconstruction method and device based on light-thunder co-location instantaneous observation

Abstract

The invention discloses a space target three-dimensional reconstruction method and device based on light-thunder co-location instantaneous observation. The method comprises the following steps: acquiring an optical image and a radar image of a space target in an optical view coordinate system; the optical image and the radar image are obtained through synchronous observation; inputting the optical image and the radar image into a pre-trained COI3F network, so that the COI3F network outputs a three-dimensional structure point cloud of the space target; the COI3F network comprises a feature generation module, a fusion module and a point cloud generation module; the feature generation module is used for generating optical image features based on the optical image and generating radar image features based on the radar image; the fusion module is used for performing feature fusion processing on the optical image features and the radar image features to obtain complete fused image features; and the point cloud generation module is used for generating a three-dimensional structure point cloud of the space target based on the complete fusion image features, so that space target three-dimensional reconstruction with high applicability, high accuracy and high timeliness is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional reconstruction of space targets, and particularly relates to a method and device for three-dimensional reconstruction of space targets based on co-located instantaneous optical and radar observations. Background Technique

[0002] With the increasing number and types of space targets year by year, the space situation has become increasingly complex, and the demand for space target detection and recognition has also increased. Optical imaging and inverse synthetic aperture radar (ISAR) two-dimensional imaging are two important means for observing space targets. However, traditional two-dimensional imaging is only the projection of the three-dimensional structure of space targets on the two-dimensional imaging plane, resulting in the compression and dimensionality reduction of the three-dimensional structure information of space targets. Three-dimensional reconstruction technology can help us understand the structure and characteristics of space targets more comprehensively, thereby improving the accuracy and efficiency of detection and recognition.

[0003] Under the co-located observation of optical and radar, the optical and radar imaging planes of space targets are orthogonal, and the observation perspectives are rich, which can combine the advantages of different sensors to complement the information of the three-dimensional image and obtain better reconstruction results. However, in order to fuse the space target characteristic information hidden in the optical and radar images under two mutually perpendicular perspectives, there is still a lack of effective processing methods.

[0004] Currently, the three-dimensional reconstruction technology of space targets is mainly divided into the following three categories. The three-dimensional reconstruction method based on optics mainly relies on the multi-view Figure 3 dimensional reconstruction principle. Among them, traditional methods can extract features and track trajectories of feature points in sequence images, and adopt template matching for typical structures such as rectangles and cylinders in key components to improve the accuracy; on the other hand, multi-view of deep networks can be adopted Figure 3Methods such as three-dimensional reconstruction, for example, generating 3D meshes from a single RGB image (A Data-Driven Approach to 3D Mesh Generation from Single RGB Images, Pixel2Mesh), multi-view stereo network (Multi-View Stereo Network, MVSNet), and Dust3R (A Novel Network for Robust 3D Reconstruction from Single RGB Images) and other networks to achieve three-dimensional reconstruction. However, optical images are easily affected by factors such as lighting and atmospheric turbulence, which affect the imaging effect. ISAR imaging is not affected by factors such as weather and mainly includes three methods: interferometric inverse synthetic aperture radar (Interferometric Inverse Synthetic Aperture Radar, InISAR), factorization, and energy accumulation. However, the hardware cost requirements for the three-dimensional reconstruction method based on InISAR are relatively high; the method based on factorization can achieve sparse point cloud reconstruction through multiple steps such as azimuth calibration, trajectory association, and removing mis-matched points; the method based on the energy accumulation method avoids the process of extracting a large number of scattering centers and trajectory association, and can jointly search for each candidate scattering point through the factorization projection relationship and the particle swarm optimization algorithm. The first two single-sensor methods have relatively high requirements for the richness of the observation angle. Therefore, the method based on multi-sensor fusion three-dimensional reconstruction has been further studied. This method uses the co-located observation images of optical and ISAR image sequences to solve the attitude information of the rotating target, improves the reconstruction integrity of the rectangular components through a semantic segmentation network with rectangular constraints, and then obtains a dense reconstruction result using a voxel-based 3D representation.

[0005] However, the above-mentioned existing three-dimensional reconstruction methods for space targets are only applicable to space targets with motion priors such as three-axis stabilization or fixed-axis slow rotation, and the reconstruction technology is difficult, and the timeliness and accuracy are low.

[0006] Therefore, how to provide a three-dimensional reconstruction method for space targets with strong applicability, high accuracy, and instantaneous reconstruction has become an urgent problem to be solved. Summary of the Invention

[0007] In order to solve the above problems existing in the prior art, the present invention provides a three-dimensional reconstruction method and device for space targets based on optical-radar co-located instantaneous observation.

[0008] The technical problems to be solved by the present invention are realized through the following technical solutions:

[0009] In the first aspect, the present invention provides a three-dimensional reconstruction method for space targets based on optical-radar co-located instantaneous observation. The three-dimensional reconstruction method for space targets includes:

[0010] Obtain an optical image and a radar image of a space target in an optical perspective coordinate system; the optical image and the radar image are obtained through synchronous observation;

[0011] Input the optical image and the radar image into a pre-trained COI3F network, so that the COI3F network outputs the three-dimensional structural point cloud of the space target;

[0012] The COI3F network includes a feature generation module, a fusion module, and a point cloud generation module;

[0013] The feature generation module is used to generate optical image features based on the optical image and generate radar image features based on the radar image;

[0014] The fusion module is used to perform feature fusion processing on the optical image features and the radar image features to obtain complete fusion image features;

[0015] The point cloud generation module is used to generate the three-dimensional structural point cloud of the space target based on the complete fusion image features.

[0016] Optionally, obtaining an optical image and a radar image of a space target in an optical perspective coordinate system includes:

[0017] Observe the space target to obtain an initial optical image and an initial radar image of the space target in a scene coordinate system;

[0018] Use a rotation matrix to perform coordinate transformation on the initial optical image and the initial radar image to obtain an optical image and a radar image in the optical perspective coordinate system.

[0019] Optionally, the fusion module is specifically used for:

[0020] Use a cross self-attention mechanism to perform feature fusion processing on the optical image features and the radar image features to obtain complete fusion image features.

[0021] Optionally, the pre-training method of the COI3F network includes:

[0022] Establish a sample set; the sample set includes multiple samples; each sample includes a sample optical image of a sample space target, a sample radar image, and a sample true model of the sample space target;

[0023] Input the sample optical images and sample radar images of each sample space target into the COI3F network, so that the COI3F network outputs the predicted three-dimensional structural point clouds of each sample space target;

[0024] Train the COI3F network according to the difference between the predicted three-dimensional structure point cloud of each sample space target and the sample real model, and use the pre-constructed loss function to train the COI3F network; the pre-constructed loss function includes a chamfer distance loss function and a model projection loss function.

[0025] Optionally, the chamfer distance loss function is:

[0026]

[0027] where d CD (·) represents the chamfer distance loss; P est represents the predicted three-dimensional structure point cloud; P real represents the sample real model; x represents the point coordinates in the predicted three-dimensional structure point cloud; y represents the point coordinates in the sample real model.

[0028] Optionally, the model projection loss function is:

[0029]

[0030] where η represents the model projection loss; I Esti-pro represents the binary image of the projection of the predicted three-dimensional structure point cloud; I Truth-pro represents the binary image of the projection of the sample real model; W represents the image width; H represents the image height; represents performing an exclusive NOR operation on the values of the binary image.

[0031] In a second aspect, the present invention provides a three-dimensional reconstruction device for a space target based on optical-radar co-located instantaneous observation. The three-dimensional reconstruction device for a space target includes:

[0032] An acquisition module, configured to acquire an optical image and a radar image of a space target in an optical perspective coordinate system; the optical image and the radar image are obtained through synchronous observation;

[0033] A three-dimensional structure point cloud generation module, configured to input the optical image and the radar image into a pre-trained COI3F network, so that the COI3F network outputs the three-dimensional structure point cloud of the space target; the COI3F network includes a feature generation module, a fusion module, and a point cloud generation module; the feature generation module is configured to generate optical image features based on the optical image and generate radar image features based on the radar image; the fusion module is configured to perform feature fusion processing on the optical image features and the radar image features to obtain complete fusion image features; the point cloud generation module is configured to generate the three-dimensional structure point cloud of the space target based on the complete fusion image features.

[0034] In a third aspect, the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus;

[0035] The memory is used to store a computer program;

[0036] The processor is used to implement the method steps described in any of the above-mentioned three-dimensional reconstruction methods of space targets based on co-located instantaneous optical and radar observations when executing the computer program stored on the memory.

[0037] In a fourth aspect, the present invention provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the method steps described in any of the above-mentioned three-dimensional reconstruction methods of space targets based on co-located instantaneous optical and radar observations are implemented.

[0038] For the three-dimensional reconstruction method of space targets based on co-located instantaneous optical and radar observations provided by the present invention, a pair of optical images and radar images obtained by synchronously observing a space target are input into a pre-trained COI3F network, and then the three-dimensional structure point cloud of the space target can be obtained, realizing the three-dimensional reconstruction of the space target, reducing the limitation of the traditional three-dimensional reconstruction target type, and having stronger applicability. Using the end-to-end COI3F network avoids the error accumulation in multi-step reconstruction and improves the accuracy of three-dimensional reconstruction; finally, through the COI3F network, instantaneous observation is realized, greatly reducing the data requirements of the traditional method, avoiding the imaging and processing problems of a large number of optical images and radar images, improving the timeliness, and realizing instantaneous reconstruction.

[0039] The following will further elaborate on the present invention in conjunction with the drawings. Description of the Drawings

[0040] Figure 1 is a schematic flowchart of a three-dimensional reconstruction method of space targets based on co-located instantaneous optical and radar observations provided by an embodiment of the present invention;

[0041] Figure 2 is a schematic structural diagram of the COI3F network provided by an embodiment of the present invention;

[0042] Figure 3 is a schematic model diagram of TG and beidou space targets;

[0043] Figure 4 is a schematic comparison diagram of the three-dimensional structure point cloud of TG reconstructed by the three-dimensional reconstruction method of space targets provided by an embodiment of the present invention and the traditional reconstruction method;

[0044] Figure 5Schematic diagram for comparing the three-dimensional structure point cloud of Beidou reconstructed by the three-dimensional reconstruction method of space targets provided in the embodiments of the present invention with the traditional reconstruction method;

[0045] Figure 6 Schematic structural diagram of a three-dimensional reconstruction device for space targets based on co-located optical and radar instantaneous observations provided in the embodiments of the present invention;

[0046] Figure 7 Schematic structural diagram of an electronic device provided in the embodiments of the present invention. Specific embodiments

[0047] The following further describes the present invention in detail with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0048] In order to solve the problems of weak applicability, large reconstruction technical difficulty, low timeliness and accuracy in the existing three-dimensional reconstruction methods of space targets, the embodiments of the present invention provide a three-dimensional reconstruction method of space targets based on co-located optical and radar instantaneous observations. See Figure 1 , Figure 1 Schematic flowchart of a three-dimensional reconstruction method of space targets based on co-located optical and radar instantaneous observations provided in the embodiments of the present invention, specifically including the following steps:

[0049] Step S101, obtain the optical image and radar image of the space target in the optical perspective coordinate system; the optical image and radar image are obtained through synchronous observation.

[0050] In the embodiments of the present invention, the optical perspective refers to the geometric space expression of the space target observed from the physical position and imaging parameters of an optical sensor (such as a camera, lidar, etc.). The radar image refers to the observation image of ISAR (Inverse-Synthetic-Aperture-Radar).

[0051] The input of the COI3F (Cross-attention with Optical and ISAR 3D Fusion) network provided in the embodiments of the present invention is usually the data directly captured by the sensor, and these data are naturally in the optical perspective. Therefore, it is necessary to convert the scene coordinate system of the space target to the optical perspective coordinate system.

[0052] In the embodiments of the present invention, the space target refers to a satellite. The optical image and radar image are a pair of co-located optical and radar images.

[0053] In one implementation, obtaining the optical image and radar image of the space target in the optical perspective coordinate system includes:

[0054] Observing a space target to obtain the initial optical image and the initial radar image of the space target in the scene coordinate system;

[0055] Using the rotation matrix to perform coordinate transformation on the initial optical image and the initial radar image to obtain the optical image and the radar image in the optical viewing coordinate system.

[0056] In the orbital coordinate system, the horizontal axis, the vertical axis, and the line-of-sight vector of the optical viewing image form a three-dimensional orthogonal coordinate system, and the three axes of this coordinate system can be expressed as where k h represents the horizontal axis, l represents the vertical axis, and k v represents the line-of-sight direction, and the superscript T represents the transpose operation of the matrix. It is necessary to transform the scene coordinate system to the optical viewing coordinate system, and the calibration value remains unchanged. Therefore, the optical viewing coordinate system O optical can be described as:

[0057] O optical = diag(norm(k h ), norm(l), norm(k v ));

[0058] where norm represents performing normalization processing; diag means creating a diagonal matrix.

[0059] The rotation matrix between the scene coordinate system and the optical viewing coordinate system can be obtained through the following calculation:

[0060] O1R optical = O optical ;

[0061]

[0062] where R optical is the rotation matrix in the optical viewing coordinate system, O1 is the scene coordinate system, represents the inverse operation of O1.

[0063] Therefore, the point cloud model of the optical viewing coordinate system can be calculated by the following formula:

[0064] P optical = R optical P;

[0065] where P is the point cloud model obtained in the scene coordinate system.

[0066] The optical imaging plane projection matrix K optical of the optical viewing coordinate system can be calculated as follows:

[0067]

[0068] Among them, is the imaging plane projection matrix of the original coordinate system.

[0069] So far, the conversion from the scene coordinate system to the optical view coordinate system is completed.

[0070] Step S102: Input the optical image and the radar image into the pre-trained COI3F network, so that the COI3F network outputs the three-dimensional structure point cloud of the spatial target. The COI3F network includes a feature generation module, a fusion module, and a point cloud generation module; the feature generation module is used to generate optical image features based on the optical image and radar image features based on the radar image; the fusion module is used to perform feature fusion processing on the optical image features and the radar image features to obtain complete fused image features; the point cloud generation module is used to generate the three-dimensional structure point cloud of the spatial target based on the complete fused image features.

[0071] In the embodiment of the present invention, the input of the COI3F network is a pair of image sets I ISAR consisting of the radar image I optical and the optical image I ISAR , I optical .

[0072] See Figure 2 , Figure 2 is the structural schematic diagram of the COI3F network provided by the embodiment of the present invention. The feature generation module further includes an inverse synthetic aperture image encoder and an optical image encoder. Since the features of the radar image and the optical image are different, different encoders need to be assigned for different image sources in order to train different weight parameters.

[0073] Perform the following transformation on the radar image. Copy the modulus values of the initial radar image to three channels respectively to form a new three-channel radar image. In this way, even if the input initial radar image I ISAR is a complex signal, the dimensions of the two input initial radar images can be kept the same:

[0074]

[0075] Among them, cat represents the splicing operation, represents the initial radar image of one channel.

[0076] Preprocess the initial radar image and the initial optical image by using the image encoding method of ViT. Among them, ViT is a model that applies the Transformer architecture to computer vision tasks. The image encoder uses the ViT method to process the image Divide it into small blocks of size 16×16, and then encode each small block into a vector with dimension D = 768 through a fully connected network. After the following processing, the 16x16 small block area I' obtained after I is divided is obtained:

[0077] I' = EmPatches(I);

[0078] Among them, I represents the initial radar image or the initial optical image, and EmPatches(I) represents the operation of dividing I into blocks. B is the number of batches; W represents the image width; H represents the image height; C represents the number of channels.

[0079] Next, construct the ViT encoder ViTEncoder(·) module. ViTEncoder(·) consists of L = 12 basic blocks. Each basic block consists of layer normalization "Norm", multi-head self-attention layer "MHAtt" and multi-layer perceptron "MLP". The l-th basic block ViTEncoderBlock l is defined as follows:

[0080] I′ l-1,mid = I′ l-1 + MHAtt(Norm(I′ l-1 ));

[0081] I′ l = MLP(Norm(I′ l-1,mid )) + I′ l-1,mid ;

[0082] ViTEncoderBlock l (I′ l-1 ) = I′ l ;

[0083] Among them, MHAtt represents the multi-head self-attention layer operation, Norm represents the layer normalization operation, MLP represents the multi-layer perceptron operation, and ViTEncoderBlock l represents the l-th basic block operation; l = 1,…L; I′ l-1,mid represents the median value of the vector of the (l - 1)-th basic block; I′ l-1 represents the image feature corresponding to the (l - 1)-th basic block; I′ l represents the image feature corresponding to the l-th basic block. For the optical image branch, ViTEncoder1 can obtain the optical image block encoding feature I'0 = I' optical , and for the radar image branch, ViTEncoder2 can obtain the radar image block encoding feature I'0 = I′ ISAR .

[0084] The separately block - encoded features of different images are respectively input into different ViTEncoders. After the image features are respectively input, different images respectively generate different feature representations, namely optical image features and radar image features as well as the position encoding of the optical image features and the position encoding of the radar image features The formula is as follows:

[0085] F optical , P optica = ViTEncoder1(I' optical );

[0086] F ISAR , P ISAR = ViTEncoder2(I I ' SAR );

[0087] Among them, F optical , that is, F1, represents the optical image features; P optica , that is, P1, represents the position encoding of the optical image features; F ISAR , that is, F2, represents the radar image features; P ISAR , that is, P2, represents the position encoding of the radar image features.

[0088] In the embodiment of the present invention, the fusion module is used to perform feature fusion processing on the optical image features and the radar image features to obtain complete fused image features.

[0089] The COI3F network fuses the features of the two branches by using the cross - self - attention mechanism "Cross - Attention" in the same image feature fusion decoder, and learns the features of the optical image and the radar image through network self - attention, so as to generate a unique three - dimensional reconstruction result under a single perspective.

[0090] The fusion module is specifically used for: using the cross - self - attention mechanism to perform feature fusion processing on the optical image features and the radar image features to obtain complete fused image features.

[0091] Specifically, it includes the following steps:

[0092] Input the optical image features and the radar image features into the COI3F initial network block. Refer to Figure 2 , input the optical image features output by the optical image encoder into the first Norm, and input the radar image features output by the inverse synthetic aperture radar image encoder into the second Norm. The decoder of COI3F can be divided into N = 8 blocks, and the processing flow of each network block is as follows:

[0093]

[0094] Among them, represents the optical image features input to the nth network block; represents the radar image features input to the nth network block; represents the optical image features input to the (n - 1)th network block; represents the radar image features input to the (n - 1)th network block; DecoderBlock i represents the decoder module in the neural network; n = 1, …, N, and the final output feature is represents the optical image features output by the Nth network block. Similarly, represents the radar image features output by the Nth network block.

[0095] Execute the cross - self - attention mechanism to fuse the image features of the two input branches. For example, Figure 2 , each network block needs to input the output result of the previous layer of the network. For each decoder block DecoderBlock, the cross - self - attention mechanism is executed to fuse the image features of the two input branches, and the specific formula process is as follows:

[0096] G2 = Norm(G2);

[0097] G1 = G1 + Drop(MHAtt(Norm(G1), P1));

[0098] G1 = G1 + Drop(CrossAttn(G1, G2, G2, P1, P2));

[0099] G1 = G1 + Drop(MLP(norm(G1)));

[0100] Among them, CrossAttn represents the cross - self - attention mechanism; G1, that is, the aforementioned represents the optical image features; G2, that is, the aforementioned represents the radar image features; "Drop" is the dropout layer, which is used in the deep network to further enhance the generalization ability of the model.

[0101] Fuse the optical image features and the radar image features in the image feature fusion decoder to output the complete fused image features.

[0102] The COI3F network has two regression heads to gradually predict the point cloud position. The first regression head adjusts the complete fused image feature G of the decoder to be consistent with the image dimension to obtain the final feature description of the entire image

[0103] X = Head1(G);

[0104] The second regression head, namely regression head 2, is used to obtain the three-dimensional structural point cloud of the complete spatial target based on X as follows:

[0105] p = Head2(X);

[0106] Where p represents the three-dimensional structural point cloud of the spatial target, Q represents the number of points in each point cloud, which can be set by oneself and will not be elaborated here.

[0107] In the embodiment of the present invention, a pair of optical images and radar images obtained by synchronously observing a spatial target are input into the pre-trained COI3F network, and then the three-dimensional structural point cloud of the spatial target can be obtained, realizing the three-dimensional reconstruction of the spatial target, reducing the limitation of the traditional three-dimensional reconstruction target type, and having stronger applicability. Using the end-to-end COI3F network can avoid the error accumulation during multi-step reconstruction and improve the accuracy of three-dimensional reconstruction; finally, through the COI3F network, instantaneous observation is realized, greatly reducing the data requirements of the traditional method, avoiding the imaging problems of a large number of optical images and radar images, improving the timeliness, and realizing instantaneous reconstruction.

[0108] In one implementation, the pre-training method of the COI3F network includes:

[0109] Establish a sample set; the sample set includes multiple samples; each sample includes a sample optical image, a sample radar image of a sample spatial target, and a sample real model of the sample spatial target;

[0110] Input the sample optical images and sample radar images of each sample spatial target into the COI3F network, so that the COI3F network outputs the predicted three-dimensional structural point cloud of each sample spatial target;

[0111] According to the difference between the predicted three-dimensional structural point cloud of each sample spatial target and the sample real model, use the pre-constructed loss function to train the COI3F network; the pre-constructed loss function includes a chamfer distance loss function and a model projection loss function.

[0112] In the embodiment of the present invention, appropriate spatial targets are selected. To ensure the generalization of the network, a large number of three-dimensional models of spatial targets are required. For example, 30 appropriate spatial targets can be selected. The 30 spatial targets can specifically include ACE, AcrimSAT, AIM, Aquarius, Aura, and CALIPSO, etc. Among them, ACE, AcrimSAT, AIM, Aquarius, Aura, and CALIPSO are all the names of satellites.

[0113] Obtain the data model of co-located observations of the ISAR and optical sensors. For each sample space target, perform 2000 co-located instantaneous observations of the optical and radar sensors, and obtain and save a pair of sample optical images and sample radar images, the observation projection matrix, and the sample target point cloud obtained from each instantaneous observation. The pair of sample optical and sample radar images are used as the input to the network, and the observation projection matrix and the sample target point cloud are used to determine the sample true model and the observation perspective.

[0114] In the embodiments of the present invention, for the conversion principle between the scene coordinate system and the optical perspective coordinate system of the sample optical image and the sample radar image, reference can be made to the foregoing content, and details are not described herein again.

[0115] In one implementation, the chamfer distance loss function includes:

[0116]

[0117] where d CD (·) represents the chamfer distance loss; P est represents the predicted three-dimensional structure point cloud; P real represents the sample true model; x represents the point coordinates in the predicted three-dimensional structure point cloud; y represents the point coordinates in the sample true model.

[0118] If the chamfer distance is larger, it indicates that the difference between the predicted three-dimensional structure point cloud and the sample true model is larger; if the chamfer distance is smaller, it indicates that the 3D reconstruction effect of the network is better.

[0119] In one implementation, a model projection loss is defined to judge the reconstruction effect. The model projection loss function is defined as follows:

[0120]

[0121] where η represents the model projection loss; I Esti-pro represents the binary image of the projection of the predicted three-dimensional structure point cloud; I Truth-pro represents the binary image of the projection of the sample true model; W represents the image width; H represents the image height; represents the exclusive NOR operation on the values of the binary image. If the value is smaller, the overlapping part of the projection is more, indicating a better reconstruction effect; otherwise, the reconstruction effect is poor.

[0122] In the embodiments of the present invention, in the direction of reducing the difference between the predicted three-dimensional structure point cloud of each sample space target and the sample true model, the COI3F network is updated using the loss function until the number of update times reaches the preset number or the function converges, and the training of the COI3F network is completed.

[0123] The simulation experiment of a three-dimensional reconstruction method for space targets based on co-located instantaneous optical and radar observations provided by the embodiments of the present invention is as follows:

[0124] See Figure 3 , Figure 3 which is a schematic diagram of the models of TG and Beidou space targets. In this simulation experiment, the TG space target model and the Beidou space target model are adopted, and the reconstruction effects and reconstruction times of the traditional method and the three-dimensional reconstruction method for space targets provided by the embodiments of the present invention are compared. The comparison data are shown in Table 1. Among them, both TG and Beidou are the names of the space target models.

[0125] Table 1

[0126]

[0127] It can be seen from Table 1 that the three-dimensional reconstruction method for space targets provided by the embodiments of the present invention, namely Ours in Table 1, has better reconstruction effects and reconstruction times than the traditional method for different space targets. The reconstructed MIou in Table 1 refers to the degree of overlap between the model prediction region and the real region, which can be simply understood as the reconstruction accuracy rate.

[0128] See Figure 4 , Figure 4 which is a schematic diagram for comparing the three-dimensional structure point clouds of TG reconstructed by the three-dimensional reconstruction method for space targets provided by the embodiments of the present invention and the traditional reconstruction method. Figure 4 In (a) of Figure 4 is the real model of TG, and in (b) of Figure 4 is the three-dimensional structure point cloud of TG reconstructed by the traditional reconstruction method under 1 pair of images, Figure 4 in (c) of Figure 4 is the three-dimensional structure point cloud of TG reconstructed by the traditional reconstruction method under 5 pairs of images, and in (d) of

[0129] in (e) of Figure 5 is the three-dimensional structure point cloud of TG reconstructed by the three-dimensional reconstruction method for space targets provided by the embodiments of the present invention under 1 pair of images. It can be seen that the accuracy of the three-dimensional structure point cloud of TG obtained by the three-dimensional reconstruction method for space targets provided by the embodiments of the present invention is higher than that of the traditional reconstruction method. Figure 5 See Figure 5 which is a schematic diagram for comparing the three-dimensional structure point clouds of Beidou reconstructed by the three-dimensional reconstruction method for space targets provided by the embodiments of the present invention and the traditional reconstruction method. Figure 5 In (a) of Figure 5Among them, (c) is the 3D structural point cloud of Beidou reconstructed by the traditional reconstruction method under 5 pairs of images. Figure 5 Among them, (d) is the 3D structural point cloud of Beidou reconstructed by the traditional reconstruction method under 10 pairs of images. Figure 5 Among them, (e) is the 3D structural point cloud of Beidou reconstructed by the 3D reconstruction method for space targets provided by the embodiments of the present invention under 1 pair of images. It can be seen that the accuracy of the 3D structural point cloud of Beidou obtained by the 3D reconstruction method for space targets provided by the embodiments of the present invention is higher than that of the traditional reconstruction method.

[0130] Based on the same inventive concept, the embodiments of the present invention also provide a 3D reconstruction device for space targets based on co-located optical and radar instantaneous observations. Refer to Figure 6 , Figure 6 which is a schematic structural diagram of a 3D reconstruction device for space targets based on co-located optical and radar instantaneous observations provided by the embodiments of the present invention. The 3D reconstruction device for space targets includes:

[0131] An acquisition module 601, configured to acquire an optical image and a radar image of a space target in an optical perspective coordinate system; the optical image and the radar image are obtained through synchronous observation;

[0132] A 3D structural point cloud generation module 602, configured to input the optical image and the radar image into a pre-trained COI3F network, so that the COI3F network outputs the 3D structural point cloud of the space target; the COI3F network includes a feature generation module, a fusion module, and a point cloud generation module; the feature generation module is configured to generate optical image features based on the optical image and generate radar image features based on the radar image; the fusion module is configured to perform feature fusion processing on the optical image features and the radar image features to obtain complete fusion image features; the point cloud generation module is configured to generate the 3D structural point cloud of the space target based on the complete fusion image features.

[0133] In the embodiments of the present invention, by inputting a pair of optical image and radar image obtained by synchronously observing a space target into a pre-trained COI3F network, the 3D structural point cloud of the space target can be obtained, realizing the 3D reconstruction of the space target, reducing the limitation of the traditional 3D reconstruction target type, and having stronger applicability. Using the end-to-end COI3F network avoids the error accumulation during multi-step reconstruction and improves the accuracy of 3D reconstruction; finally, through the COI3F network, instantaneous observation is realized, greatly reducing the data requirements of the traditional method, avoiding the imaging and processing problems of a large number of optical images and radar images, improving the timeliness, and realizing instantaneous reconstruction.

[0134] Optionally, the acquisition module is specifically configured to:

[0135] Observing a space target to obtain an initial optical image and an initial radar image of the space target in a scene coordinate system; using a rotation matrix to perform coordinate transformation on the initial optical image and the initial radar image to obtain an optical image and a radar image in an optical view coordinate system.

[0136] Optionally, the fusion module is specifically configured to:

[0137] Using a cross self-attention mechanism to perform feature fusion processing on the optical image features and the radar image features to obtain complete fused image features.

[0138] Optionally, the pre-training method of the COI3F network includes:

[0139] Establishing a sample set; the sample set includes multiple samples; each sample includes a sample optical image of a sample space target, a sample radar image, and a sample true model of the sample space target;

[0140] Inputting the sample optical images and sample radar images of each sample space target into the COI3F network so that the COI3F network outputs the predicted three-dimensional structure point clouds of each sample space target;

[0141] Training the COI3F network according to the difference between the predicted three-dimensional structure point clouds of each sample space target and the sample true model; the pre-constructed loss function includes a chamfer distance loss function and a model projection loss function.

[0142] Optionally, the chamfer distance loss function is:

[0143]

[0144] where d CD (·) represents the chamfer distance loss; P est represents the predicted three-dimensional structure point cloud; P real represents the sample true model; x represents the point coordinates in the predicted three-dimensional structure point cloud; y represents the point coordinates in the sample true model.

[0145] Optionally, the model projection loss function is:

[0146]

[0147] where η represents the model projection loss; I Esti-pro represents the binary image of the projection of the predicted three-dimensional structure point cloud; I Truth-pro represents the binary image of the projection of the sample true model; W represents the image width; H represents the image height; represents performing an exclusive NOR operation on the values of the binary image.

[0148] An embodiment of the present invention further provides an electronic device, such as Figure 7 shown, which includes a processor 701, a communication interface 702, a memory 703, and a communication bus 704. Among them, the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0149] The memory 703 is used to store a computer program.

[0150] When the processor 701 is used to execute the program stored on the memory 703, the method steps of any one of the above-mentioned three-dimensional reconstruction methods of spatial targets based on co-located optical and radar instantaneous observations are implemented.

[0151] The communication bus mentioned in the above electronic device may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, only a thick line is used in the figure to represent it, but it does not mean that there is only one bus or one type of bus.

[0152] The communication interface is used for communication between the above electronic device and other devices.

[0153] The memory may include a Random Access Memory (RAM), or may also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.

[0154] The above-mentioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0155] The present invention also provides a computer-readable storage medium. A computer program is stored in the computer-readable storage medium. When the computer program is executed by a processor, the method steps of any one of the above-mentioned three-dimensional reconstruction methods of a space target based on co-located instantaneous optical and radar observations are implemented.

[0156] Optionally, the computer-readable storage medium may be a non-volatile memory (Non-Volatile Memory, NVM), for example, at least one disk memory.

[0157] Optionally, the above-mentioned computer-readable storage medium may also be at least one storage device located away from the aforementioned processor.

[0158] In another embodiment of the present invention, a computer program product containing instructions is also provided. When it runs on a computer, it causes the computer to execute the method steps of any one of the above-mentioned three-dimensional reconstruction methods of a space target based on co-located instantaneous optical and radar observations.

[0159] It should be noted that the terms "first", "second", etc. are used to distinguish similar objects and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of the present invention described here can be implemented in an order different from those illustrated or described here. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present invention.

[0160] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. means that the specific features or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification.

[0161] Although the present invention has been described in connection with various embodiments, those skilled in the art can understand and implement other variations of the disclosed embodiments by viewing the accompanying drawings and the disclosure during the implementation of the claimed invention. In the description of the present invention, the term "including" does not exclude other components or steps, the use of "a" or "an" does not exclude a plurality of cases, and the meaning of "plurality" is two or more, unless otherwise specifically defined. In addition, certain measures are described in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0162] The method provided by the embodiments of the present invention can be applied to electronic devices. Specifically, the electronic device can be: a desktop computer, a portable computer, a smart mobile terminal, a server, etc. There is no limitation here, and any electronic device that can implement the present invention belongs to the protection scope of the present invention.

[0163] For the device / electronic device / storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0164] It should be noted that the device, electronic device, and storage medium of the embodiments of the present invention are respectively the device, electronic device, and storage medium applying the above-mentioned method for three-dimensional reconstruction of a space target based on co-located instantaneous optical and radar observations. Then all embodiments of the above-mentioned method for three-dimensional reconstruction of a space target based on co-located instantaneous optical and radar observations are applicable to the device, electronic device, and storage medium, and can achieve the same or similar beneficial effects.

[0165] The above content is a further detailed description of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is only limited to these descriptions. For those of ordinary skill in the technical field to which the present invention belongs, without departing from the concept of the present invention, several simple deductions or substitutions can still be made, and all should be regarded as belonging to the protection scope of the present invention.

Claims

1. A method for three-dimensional reconstruction of space targets based on instantaneous observation of light-radar co-location, characterized in that: The space target three-dimensional reconstruction method comprises: Acquire an optical image and a radar image of a space target in an optical viewing angle coordinate system; the optical image and the radar image are obtained by synchronous observation; Inputting the optical image and the radar image into a pre-trained COI3F network so that the COI3F network outputs a three-dimensional structure point cloud of the space target; The COI3F network includes a feature generation module, a fusion module and a point cloud generation module; The feature generation module is used to generate an optical image feature based on the optical image and to generate a radar image feature based on the radar image; The fusion module is used to perform feature fusion processing on the optical image features and the radar image features to obtain complete fused image features; The point cloud generation module is used to generate a three-dimensional structure point cloud of the space target based on the complete fusion image features.

2. The method for three-dimensional reconstruction of a space target according to claim 1, characterized in that: Obtain optical images and radar images of space targets in the optical view coordinate system, including: Observe the space target to obtain an initial optical image and an initial radar image of the space target in a scene coordinate system; The initial optical image and the initial radar image are transformed into coordinate systems by using a rotation matrix to obtain an optical image and a radar image in an optical viewing angle coordinate system.

3. The method for three-dimensional reconstruction of a space target according to claim 1, characterized in that: The fusion module is specifically used for: The optical image features and the radar image features are fused using a cross self-attention mechanism to obtain complete fused image features.

4. The method for three-dimensional reconstruction of a space target according to claim 1, characterized in that: The pre-training method of the COI3F network includes: Establishing a sample set; the sample set includes a plurality of samples; each sample includes a sample optical image of a sample space target, a sample radar image and a sample real model of the sample space target; Inputting the sample optical image and the sample radar image of each sample space target into the COI3F network so that the COI3F network outputs the predicted three-dimensional structure point cloud of each sample space target; According to the difference between the predicted three-dimensional structure point cloud of each sample space target and the real model of the sample, the COI3F network is trained using a pre-constructed loss function; the pre-constructed loss function includes a chamfer distance loss function and a model projection loss function.

5. The method for three-dimensional reconstruction of a space target according to claim 4, characterized in that: The chamfer distance loss function is: Among them, d CD (·) indicates chamfer distance loss; P est Represents the predicted three-dimensional structure point cloud; P real represents the true model of the sample; x represents the point coordinates in the predicted 3D structure point cloud; y represents the point coordinates in the true model of the sample.

6. The method for three-dimensional reconstruction of a space target according to claim 4, characterized in that: The model projection loss function is: Where η represents the model projection loss; I Esti-pro I represents the binary image of the predicted 3D structure point cloud projection; Truth-pro Represents the binary image of the sample's true model projection; W represents the image width; H represents the image height; Indicates the XOR operation on the values ​​of the binary image.

7. A three-dimensional reconstruction device for space targets based on light-radar co-location instantaneous observation, characterized in that: The space target three-dimensional reconstruction device comprises: An acquisition module, used to acquire an optical image and a radar image of a space target in an optical viewing angle coordinate system; the optical image and the radar image are acquired by synchronous observation; A three-dimensional structure point cloud generation module is used to input the optical image and the radar image into a pre-trained COI3F network so that the COI3F network outputs a three-dimensional structure point cloud of the space target; the COI3F network includes a feature generation module, a fusion module and a point cloud generation module; the feature generation module is used to generate optical image features based on the optical image, and to generate radar image features based on the radar image; the fusion module is used to perform feature fusion processing on the optical image features and the radar image features to obtain complete fused image features; the point cloud generation module is used to generate a three-dimensional structure point cloud of the space target based on the complete fused image features.

8. The space target three-dimensional reconstruction device according to claim 7, characterized in that: The acquisition module is specifically used for: The space target is observed to obtain an initial optical image and an initial radar image of the space target in a scene coordinate system; and the initial optical image and the initial radar image are transformed into coordinate systems using a rotation matrix to obtain an optical image and a radar image in an optical viewing angle coordinate system.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the three-dimensional reconstruction method of a space target as described in any one of claims 1 to 6 when executing a computer program stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for three-dimensional reconstruction of a space target according to any one of claims 1 to 6 is implemented.

Images