A spatial non-cooperative target depth completion method based on geometric primitive constraints

Abstract

This invention relates to a depth completion method for non-cooperative spatial targets based on geometric primitive constraints, belonging to the field of 3D structure reconstruction technology for spatial targets. The method includes: establishing local depth constraint equations based on the target's geometric primitives; constructing a multi-task depth completion neural network model and incorporating the local depth constraint equations into the network loss function; establishing a sample dataset and training the multi-task depth completion neural network model; and embedding the trained model into the information processing unit of a spaceborne platform for target depth and pose prediction. This method addresses the problems of low accuracy and interpretability in the fine structure recovery of existing neural network-based target depth completion models, offering advantages such as high accuracy, fast speed, and strong interpretability of prediction results, thus achieving accurate recovery of the 3D fine structure of spatial targets under structural constraints.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional structure reconstruction technology for space targets, and specifically to a depth completion method for non-cooperative space targets based on geometric primitive constraints. Background Technology

[0002] With the rapid development of aerospace technology, an increasing number of satellites are being launched into space. Some of these satellites, after running out of fuel or malfunctioning, lose control and become space debris drifting freely in space. This space debris not only occupies valuable orbital resources but also seriously threatens the operational safety of satellites in orbit. Currently, tasks such as rapid removal of space debris, on-orbit repair of malfunctioning satellites, and on-orbit refueling of spacecraft have become important directions for the development of aerospace technology. These missions require the use of visual navigation systems to acquire real-time target structure and attitude information. However, the lack of communication and uncooperative maneuvering behavior among non-cooperative targets presents significant challenges to on-orbit missions targeting non-cooperative targets.

[0003] The target depth completion task based on visible light cameras and LiDAR aims to reconstruct pixel-level depth information from sparse ranging information acquired by LiDAR, guided by visible light images. Due to its important applications in various tasks, this technology has attracted widespread attention from numerous scholars. Early research utilized traditional image processing techniques (such as guided filtering and global optimization) to generate dense depth images. In recent years, the powerful feature extraction capabilities of neural networks have enabled learning-based depth completion methods to outperform traditional methods in both accuracy and efficiency. Although deep learning-based depth completion methods have improved the accuracy and efficiency of depth prediction to some extent, they largely rely on data-driven models and the powerful nonlinear modeling capabilities of neural networks to recover target depth information. They lack explicit modeling of the target's geometric structural features, resulting in limited accuracy in recovering the target's fine structure and low model interpretability. Therefore, a spatial non-cooperative target depth completion method based on geometric primitive constraints is urgently needed to address these issues. Summary of the Invention

[0004] The purpose of this invention is to provide a spatial non-cooperative target depth completion method based on geometric primitive constraints. This method aims to overcome the problem that existing depth completion neural network models lack explicit modeling of the geometric structural features of the target, resulting in limited accuracy in restoring the fine structure of the target and low interpretability of the model. The invention achieves adaptive association between target depth and geometric primitives, thereby improving the accuracy and interpretability of target structure restoration.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: a spatial non-cooperative target depth completion method based on geometric primitive constraints, comprising the following steps: Establish local depth constraint equations for the primitives based on the target geometric primitives; A multi-task deep completion neural network model is constructed, and the local depth constraint equation of the primitives is introduced into the network loss function; Establish a sample dataset and train a multi-task deep completion neural network model; The trained multi-task deep completion neural network model is embedded into the information processing unit of the spaceborne platform for target depth and pose prediction.

[0006] Preferably, based on the analysis of typical spacecraft configuration features, a target geometric primitive parameterized model library is constructed, and a primitive local depth constraint equation is established in combination with the camera imaging model. The typical spacecraft configuration features include cylindrical, spherical, planar, conical, and ellipsoidal surfaces.

[0007] Preferably, the local depth constraint equation of the primitive is expressed as: ; in, Let be the coordinates of the primitive point in the camera coordinate system; This represents the primitive category identifier corresponding to the i-th primitive instance. Let represent the primitive parameter vector corresponding to the i-th primitive instance. This represents the pixel coordinates of the primitive in the camera.

[0008] Preferably, the network loss function is mainly composed of a weighted average of pixel depth prediction loss and primitive parameter depth constraint loss, wherein the primitive parameter depth constraint loss embeds a primitive local depth constraint equation, expressed as: ; in, pixel coordinates Based on predicted depth The spatial coordinates in the camera coordinate system obtained by reverse calculation This is a probability graph of the predicted primitive types. These are the predicted geometric primitive parameters.

[0009] Preferably, a three-dimensional model of the space target is collected, the parameters of the optical camera and lidar sensor and the on-orbit simulation conditions are set, and the optical image, sparse depth image and dense depth ground label of the target are acquired to establish a sample dataset.

[0010] Preferably, the multi-task deep completion neural network model includes a feature encoder, a feature fusion module, and a multi-branch decoder; the feature encoder is used to extract target optical features and sparse depth features from the target optical image and sparse depth image; the feature fusion module is used to fuse the target optical features and sparse depth features; and the multi-branch decoder obtains pixel primitive type, primitive parameters, and depth information based on the fused features.

[0011] Preferably, the multi-task deep completion neural network model has two parallel feature encoders. Each feature encoder uses a backbone network with deformable convolutions. On the regular sampling grid of standard convolutions, a two-dimensional offset learned by the network is added to each sampling point, so that the convolution kernel can adaptively adjust the sampling position.

[0012] Preferably, the feature fusion module performs the following operation to fuse target optical features and sparse depth features, including: Statistically analyze the mean and variance of the optical and sparse depth features of the target at different spatial locations; The statistical results are concatenated and then convolved in 3×3. The convolution result is then input into the Sigmoid function for activation to obtain the optical feature importance map at each spatial location. Based on the importance map of optical features, the optical features and depth features at the same level are weighted and summed to obtain the fused features.

[0013] Preferably, the multi-branch decoder includes three parallel decoding branches: a primitive class segmentation branch, a primitive parameter estimation branch, and a dense depth regression branch. Each branch uses a U-Net decoding structure to upsample and refine the fused features, and outputs the following results: Primitive category splitting branch outputs a dimension of Primitive type probability graph Where H and W are the height and width of the image, The number of predefined geometric primitive types, where 1 represents background or other primitive categories; The primitive parameter estimation branch outputs the geometric primitive parameters corresponding to each pixel. ;in, The maximum number of parameters required to describe all possible primitives; Represents a three-dimensional real tensor space, where each pixel position corresponds to a value belonging to the tensor space. The parameter vector represents the parameter values ​​of the geometric primitive corresponding to the pixel; The dense depth regression branch outputs the predicted depth for each pixel. .

[0014] Preferably, the sample dataset is divided into a training set and a validation set; the multi-task deep completion neural network model is trained and optimized based on the training set, and the model parameters are updated. When the model parameters are updated a set number of times, the model performance is verified using the validation set sample data, and the model parameters with the best performance are used as the parameters of the multi-task deep completion neural network model after training.

[0015] Beneficial effects: The method of this invention establishes a local depth constraint equation based on the geometric primitives of the target and embeds it into the network loss function of a multi-task depth completion neural network model. It makes full use of the geometric structural features of the primitives and the relationship between camera imaging depth constraints, thereby improving the accuracy of dense depth recovery of spatial targets. It can solve the problems of low accuracy and low interpretability of fine structure recovery in existing neural network-based target depth completion models. It has the advantages of high accuracy, fast speed, and strong interpretability of prediction results, and realizes accurate recovery of the three-dimensional fine structure of spatial targets under structural constraints. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the spatial non-cooperative target depth completion method based on geometric primitive constraints according to the present invention. Figure 2 This is a schematic diagram of typical spacecraft configuration features in this invention; Figure 3 This is the overall block diagram of the multi-task deep completion neural network model in this invention. Detailed Implementation

[0017] To make the objectives and advantages of this invention clearer, the invention will be specifically described below with reference to embodiments. It should be understood that the following text is merely used to describe one or more specific embodiments of the invention and does not strictly limit the scope of protection specifically claimed by the invention.

[0018] Examples, such as Figure 1 As shown, a spatial non-cooperative target depth completion method based on geometric primitive constraints includes the following steps: Analyzing typical spacecraft configuration characteristics, a parameterized model library of target geometric primitives is constructed. Local depth constraint equations for these primitives are established in conjunction with camera imaging models, referencing... Figure 2 As shown, considering that typical spacecraft configurations have significant artificial object characteristics, their complex shapes can be decomposed into a combination of several regular geometric primitives with analytical expressions. Typical spacecraft primitives include planes (solar panels), cones (engine nozzles), spheres (fuel tanks), and parabolic surfaces (communication antennas), etc. Based on this, this embodiment constructs a parametric geometric representation model library containing typical primitives such as planes, cylinders, spheres, cones, and ellipsoids, and establishes a local depth constraint paradigm for each primitive in the camera observation coordinate system. Assume the target consists of multiple primitive instances. The composition, and the primitive categories and primitive parameters corresponding to each primitive instance are as follows: and The analytic expression for the primitive surface can be represented as: ; in, primitive instance The coordinates of a point in the camera coordinate system; based on the pinhole camera model, the coordinates... Its corresponding pixel coordinates The projection relationship can be expressed as: ; in and For camera internal parameters; and Representing the image coordinate system respectively direction and Focal length parameters of direction, and The principal point coordinates are represented separately. Substituting the above equation into the primitive parameterization model, we obtain the primitive local depth constraint equation, i.e., the pixel position. Depth Z and primitive parameters satisfy: ; A multi-task deep completion neural network model is designed to simultaneously achieve target primitive segmentation, primitive parameter estimation, and pixel depth regression. A local depth constraint equation for primitives is introduced into the network loss function, which includes pixel depth prediction loss and primitive parameter depth constraint loss. The pixel depth prediction loss aims to minimize the deviation between the depth image prediction result and the ground label, while the primitive parameter depth constraint loss aims to constrain the predicted primitive type, the geometric projection relationship between the primitive parameters and the predicted depth. By explicitly embedding the geometric structure regularity of the primitives into the depth estimation process, the completion result maintains the continuity of local details while better conforming to the global geometric rationality of the 3D scene.

[0019] refer to Figure 3 As shown, the multi-task deep completion neural network model includes a feature encoder, a feature fusion module, and a multi-branch decoder. The feature encoder is used to extract the target optical features and sparse depth features from the target optical image and sparse depth image. The feature fusion module is used to fuse the target optical features and sparse depth features. The multi-branch decoder obtains the pixel primitive type, primitive parameters, and depth information based on the fused features.

[0020] A 3D model of the spatial target is collected, and imaging parameters of the optical camera and lidar sensor, as well as working parameters such as illumination direction, target distance, and target attitude are set. Optical images, sparse depth images, and dense depth ground label data of the target are generated using the 3D model of the target and simulation software to establish a sample dataset, which is divided into a training set and a validation set. The parameters of the multi-task depth completion neural network model are trained and optimized using the training set data. After the network parameters are updated a set number of times, the model performance is verified using the validation set data. The model parameters corresponding to the optimal model performance are saved as the trained multi-task depth completion neural network model. The saved model parameters are used as the final model weights. Test data is loaded into the model to obtain the depth completion results, and the network accuracy is evaluated by comparing them with the ground labels.

[0021] In one embodiment, the multi-task deep completion neural network model includes two parallel feature encoders. Each feature encoder employs a backbone network with deformable convolutions (such as ResNet or a similar structure with deformable convolutions). On the regular sampling grid of standard convolutions, a two-dimensional offset learned by the network is added to each sampling point, enabling the convolution kernel to adaptively focus on the key geometric contours and structural regions of the target, thereby enhancing the feature extraction capability for complex shapes of spacecraft (such as the edges of solar panels, cylindrical thrusters, ellipsoidal antennas, etc.).

[0022] In one embodiment, the multi-branch decoder includes three parallel decoding branches: a primitive class segmentation branch, a primitive parameter estimation branch, and a dense depth regression branch.

[0023] In one specific embodiment, the target optical image and sparse depth images The inputs are fed into two parallel feature encoders; after multiple levels of downsampling, the optical encoder and depth encoder output multi-level optical feature maps respectively. With depth feature map ,in , , ; feature map and Input is fed into the feature fusion module, and output is the fused feature pyramid. ;for , First, the mean and variance of the target's optical features and sparse depth features at different spatial locations are statistically analyzed. The statistical results are then concatenated and subjected to a 3×3 convolution. The convolution result is then input into the Sigmoid function for activation to obtain the importance map of optical features at each spatial location; specifically, this can be represented as: ; in, Indicates the dimension along the channel Perform global average pooling on the input feature map; Indicates the dimension along the channel Perform standard deviation statistics on the input feature map; Indicates the kernel size as Convolution operation; express Activation function, used to map the output to to The range of values ​​is used to obtain the importance weight of optical features at each spatial location.

[0024] Based on the importance map of optical features, a weighted summation of optical features and depth features at the same level is performed to obtain the fused feature, which is represented as: ; Among them, symbols This represents element-wise multiplication. Used as optical features Location weight map Used as a deep feature The complementary position weight map is used to achieve position-wise weighted summation of the two types of features.

[0025] In the feature decoding stage, the feature pyramid will be fused. The input is fed into a decoder with three parallel branches, each branch responsible for a specific prediction task: Primitive category segmentation branch: This branch is a semantic segmentation network that uses a U-Net decoding structure to upsample and refine the fused features, ultimately outputting a dimension... Primitive type probability graph ,in The number of predefined geometric primitive types, The additional 1 in "" represents the background or other primitive category; for each pixel location in the image , It is A dimensional vector represents the probability that a pixel belongs to any type of primitive or the background; by taking the index of the highest probability, the predicted primitive category label for each pixel can be obtained.

[0026] Primitive Parameter Estimation Branch: The primitive parameter estimation branch also uses the U-Net decoding structure to upsample and refine the fused features, outputting the geometric primitive parameters corresponding to each pixel. ;in This is the maximum number of parameters required to describe all possible primitives; for a given pixel... If its predicted category is the first Class primitives, then from Extract the first The specific number of parameters required for class primitives.

[0027] Dense Depth Regression Branch: This branch also uses the U-Net decoding structure to upsample and refine the fused features, outputting the predicted depth for each pixel. ; The pixel depth prediction loss is weighted by the primitive parameter depth constraint loss, and expressed as: ; in , The weights for pixel depth prediction loss and primitive parameter depth constraint loss are respectively: pixel depth prediction loss Using L1 loss to directly measure the predicted depth map Dense depth map with ground truth The difference in effective pixel positions is represented as: ; in, It is the set of pixels that are effective for ground truth depth. This refers to the number of effective pixels. The primitive parameter depth constraint loss incorporates the primitive local depth constraint equation, expressed as: ; In the formula, pixel position Based on predicted depth The spatial coordinates in the camera coordinate system obtained by reverse calculation and The primitive category segmentation branch and the primitive parameter estimation branch are respectively in The prediction results at that location.

[0028] After training, the saved model parameters are solidified and embedded into the information processing unit of the spaceborne platform. During on-orbit operation, the spaceborne optical camera and lidar acquire target optical images and sparse depth maps in real time, which are then input into the multi-task depth completion neural network model for forward inference. The model simultaneously outputs the target dense depth, primitive category, and primitive parameter prediction results, providing high-precision and highly interpretable input data for subsequent target 3D reconstruction, pose estimation, and other tasks.

[0029] In summary, the method of this invention establishes a parameterized model library of geometric primitives, including typical geometric primitives such as cylinders, spheres, planes, cones, and ellipsoids, based on the typical configuration characteristics of spatial targets. It also derives the primitive parameter and pixel depth constraint equations in conjunction with the camera imaging model. Furthermore, it designs a multi-task depth completion neural network model for primitive segmentation, primitive parameter estimation, and pixel depth regression, and incorporates the primitive parameter and pixel depth constraint equations into the loss function. This ensures that the network prediction results conform to the relationship between the primitive geometric structure characteristics and the camera imaging depth constraints, thereby improving the accuracy and interpretability of target depth prediction.

[0030] The embodiments of the present invention have been described in detail above with reference to the examples. However, the present invention is not limited to the above embodiments. For those skilled in the art, after learning the contents described in the present invention, several equivalent changes and substitutions can be made without departing from the principle of the present invention. These equivalent changes and substitutions should also be considered to fall within the protection scope of the present invention.

Claims

1. A spatial non-cooperative target depth completion method based on geometric primitive constraints, characterized in that: Includes the following steps: Establish local depth constraint equations for the primitives based on the target geometric primitives; A multi-task deep completion neural network model is constructed, and the local depth constraint equation of the primitives is introduced into the network loss function. Establish a sample dataset and train a multi-task deep completion neural network model; The trained multi-task deep completion neural network model is embedded into the information processing unit of the spaceborne platform for target depth and pose prediction.

2. The spatial non-cooperative target depth completion method based on geometric primitive constraints according to claim 1, characterized in that: Based on the analysis of typical spacecraft configuration features, a target geometric primitive parameterized model library is constructed, and a primitive local depth constraint equation is established in combination with the camera imaging model. The typical spacecraft configuration features include cylindrical, spherical, planar, conical, and ellipsoidal surfaces.

3. The spatial non-cooperative target depth completion method based on geometric primitive constraints according to claim 2, characterized in that: The local depth constraint equation of the primitive is expressed as: ； in, Let be the coordinates of the primitive point in the camera coordinate system; This represents the primitive category identifier corresponding to the i-th primitive instance. Let represent the primitive parameter vector corresponding to the i-th primitive instance. This represents the pixel coordinates of the primitive in the camera.

4. The spatial non-cooperative target depth completion method based on geometric primitive constraints according to claim 3, characterized in that: The network loss function is mainly composed of a weighted average of pixel depth prediction loss and primitive parameter depth constraint loss, wherein the primitive parameter depth constraint loss embeds a primitive local depth constraint equation, expressed as: ； in, pixel coordinates Based on predicted depth The spatial coordinates obtained in the camera coordinate system by reverse calculation This is a probability graph of the predicted primitive types. These are the predicted geometric primitive parameters.

5. The spatial non-cooperative target depth completion method based on geometric primitive constraints according to claim 1, characterized in that: Collect 3D models of space targets, set parameters for optical cameras and lidar sensors and on-orbit simulation conditions, and acquire optical images, sparse depth images and dense depth ground labels of targets to establish a sample dataset.

6. The spatial non-cooperative target depth completion method based on geometric primitive constraints according to claim 5, characterized in that: The multi-task deep completion neural network model includes a feature encoder, a feature fusion module, and a multi-branch decoder; the feature encoder is used to extract target optical features and sparse depth features from target optical images and sparse depth images. The feature fusion module is used to fuse target optical features and sparse depth features; The multi-branch decoder obtains pixel primitive type, primitive parameters, and depth information based on fused features.

7. The spatial non-cooperative target depth completion method based on geometric primitive constraints according to claim 6, characterized in that: The multi-task deep completion neural network model has two parallel feature encoders. Each feature encoder uses a backbone network with deformable convolution. On the regular sampling grid of standard convolution, a two-dimensional offset learned by the network is added to each sampling point, so that the convolution kernel can adaptively adjust the sampling position.

8. The spatial non-cooperative target depth completion method based on geometric primitive constraints according to claim 6, characterized in that: The feature fusion module performs the following operations to fuse the target optical features and sparse depth features, including: Statistically analyze the mean and variance of the optical and sparse depth features of the target at different spatial locations; The statistical results are concatenated and then convolved in 3×3. The convolution result is then input into the Sigmoid function for activation to obtain the optical feature importance map at each spatial location. Based on the importance map of optical features, the optical features and depth features at the same level are weighted and summed to obtain the fused features.

9. The spatial non-cooperative target depth completion method based on geometric primitive constraints according to claim 6, characterized in that: The multi-branch decoder includes three parallel decoding branches: a primitive class segmentation branch, a primitive parameter estimation branch, and a dense depth regression branch. Each branch uses the U-Net decoding structure to upsample and refine the fused features, and outputs the following results: Primitive category splitting branch outputs a dimension of Primitive type probability graph Where H and W are the height and width of the image, The number of predefined geometric primitive types, where 1 represents background or other primitive categories; The primitive parameter estimation branch outputs the geometric primitive parameters corresponding to each pixel. ;in, The maximum number of parameters required to describe all possible primitives; Represents a three-dimensional real tensor space, where each pixel position corresponds to a value belonging to the tensor space. The parameter vector represents the parameter values ​​of the geometric primitive corresponding to the pixel; The dense depth regression branch outputs the predicted depth for each pixel. .

10. The spatial non-cooperative target depth completion method based on geometric primitive constraints according to claim 5, characterized in that: The sample dataset is divided into a training set and a validation set. The multi-task deep completion neural network model is trained and optimized based on the training set, and the model parameters are updated. After the model parameters have been updated a set number of times, the model performance is verified using the validation set sample data. The model parameters with the best performance are used as the parameters of the multi-task deep completion neural network model after training.

Images