Spacecraft Depth Completion Method Based on Temporal Optical Images and LiDAR Data

Abstract

A method for spacecraft depth completion based on temporal optical images and lidar data, belonging to the technical field of three-dimensional structure perception of space targets. The present invention aims at the problem that existing single-frame target depth completion methods ignore the time-related information of consecutive frames, resulting in inconsistent target temporal depth completion results. It includes obtaining a temporal sequence of target gray-scale images of a monocular visible light camera and a temporal sequence of sparse depth images of a lidar based on the simulation of a three-dimensional model of a space target, and generating target dense depth ground truth labels for the images; using the temporal sequence of target gray-scale images and the temporal sequence of sparse depth images as samples to train a temporal spacecraft depth completion neural network model, obtaining a trained temporal spacecraft depth completion neural network model and embedding it into an on-board platform, and realizing the prediction of dense depth of space targets based on real-time sensing data collected by a monocular visible light camera and a lidar. The present invention is used for depth completion in spacecraft detection.

Description

Technical Field

[0001] The present invention relates to a spacecraft depth completion method based on temporal optical images and lidar data, and belongs to the technical field of three-dimensional structure perception of space targets. Background Art

[0002] With the rapid development of space technology, the types of human space on-orbit missions are gradually showing a diversified development trend. Among them, perceiving the three-dimensional structure of spacecraft and obtaining its point cloud data are the prerequisites for the successful execution of many space on-orbit missions, such as debris cleaning, on-orbit maintenance, rendezvous and docking, etc.

[0003] Currently, the mainstream three-dimensional structure perception schemes for space targets can be mainly divided into stereo vision systems, time-of-flight (TOF) cameras, and combinations of monocular cameras and lidars. Among them, stereo vision systems use the triangulation principle to restore the depth of the extracted feature points, and have poor effects on objects with smooth surfaces or repetitive textures; in addition, the baseline of binocular cameras greatly limits the working distance of the system, making it difficult to meet the requirements of space on-orbit missions. TOF cameras calculate the precise depth of the target by measuring the time delay between the emitted and received laser pulses. Although precise depth can be obtained at high density, due to on-orbit power limitations, the working distance of TOF cameras is usually less than 10m, hindering their use in practical applications. Monocular cameras and lidars have a longer working distance and use optical images and sparse ranging information to restore the dense depth of spacecraft. Compared with binocular systems and TOF cameras, combining monocular cameras and lidars can effectively increase the working distance of the system and reduce the sensitivity to lighting conditions and materials, making it more suitable for use in space practical applications.

[0004] Since the realization of dense depth recovery of targets based on monocular cameras and lidars has important applications in many scenarios, a large number of target depth completion algorithms based on deep learning have been proposed in recent years to meet the various depth-based application requirements. Although important progress has been made in the problem of target depth completion based on single-frame sensor data, the data that needs to be processed during actual on-orbit work is sensor sequence data. Existing single-frame target depth completion methods ignore the time-related information of consecutive frames, resulting in inconsistent target temporal depth completion results. Therefore, the present invention aims to study a space spacecraft depth completion method based on sequence data. Summary of the Invention

[0005] Aiming at the problem that existing single-frame target depth completion methods ignore the time-related information of consecutive frames, resulting in inconsistent target temporal depth completion results, the present invention provides a spacecraft depth completion method based on temporal optical images and lidar data.

[0006] A method for spacecraft depth completion based on sequential optical images and lidar data according to the present invention includes:

[0007] Collect multiple three-dimensional models of space targets, set the simulation working conditions of the three-dimensional models of space targets and the sensor parameters of the monocular visible light camera and lidar; based on the simulation of multiple three-dimensional models of space targets, obtain the sequential gray-scale image sequence of the target of the monocular visible light camera and the sequential sparse depth image sequence of the lidar, and generate the target dense depth ground truth of the images.

[0008] Construct a sequential spacecraft depth completion neural network model: including multiple target depth prediction branches cascaded in chronological order.

[0009] Each target depth prediction branch includes an encoding stage and a decoding stage.

[0010] The encoding stage includes a foreground segmentation module, a gray-scale image feature extraction module, a morphological preprocessing module, and a depth image feature extraction module; the decoding stage includes an LSTM module and a deconvolution layer.

[0011] The prediction process of the space target depth at time t includes:

[0012] The sparse depth image I at time t s,t After being preprocessed by the morphological preprocessing module, the preprocessed depth image is input into the depth image feature extraction module.

[0013] The target gray-scale image I at time t g,t The multi-scale gray-scale image feature maps of different semantic levels and different resolutions are extracted by the gray-scale image feature extraction module.

[0014] For the sparse depth image I at time t s,t and the target gray-scale image I g,t After cascade operation, they are input into the foreground segmentation module for target foreground segmentation to obtain the foreground segmentation image.

[0015] The depth image feature extraction module first processes the preprocessed depth image through a convolutional layer, and then performs feature fusion with the gray-scale image feature maps level by level in the order from large to small in scale, to obtain the multi-modal data fusion feature F at time t t ;

[0016] The multi-modal data fusion feature F at time t t Performs temporal relationship modeling with the corresponding hierarchical feature states at time t-1 through the LSTM module level by level, and performs feature decoding through the deconvolution layer; the feature decoding result of the last layer is combined with the foreground segmentation image to obtain the target depth prediction result at time t.

[0017] Using the time - series of target grayscale images from a monocular visible - light camera and the time - series of sparse depth images from a lidar as sample images to train a time - series spacecraft depth completion neural network model, obtaining the trained time - series spacecraft depth completion neural network model;

[0018] Embedding the trained time - series spacecraft depth completion neural network model into an on - board platform, and realizing the prediction of dense depth of space targets based on the real - time sensing data collected by the monocular visible - light camera and the lidar.

[0019] According to the spacecraft depth completion method based on time - series optical images and lidar data of the present invention, for the target grayscale image I g,t The grayscale image feature extraction module extracts grayscale image feature maps of five sizes, and the sizes are 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the target grayscale image I g,t respectively.

[0020] According to the spacecraft depth completion method based on time - series optical images and lidar data of the present invention, the process of the depth image feature extraction module obtaining the multi - modal data fusion feature F t at time t includes:

[0021] The depth image feature extraction module first processes the pre - processed depth image through a convolutional layer, then processes it through a first - level residual module and fuses it with the 1 / 2 grayscale image feature map using a first - level feature fusion module to obtain a first - level data fusion feature; the first - level data fusion feature is processed through a second - level residual module and fused with the 1 / 4 grayscale image feature map using a second - level feature fusion module to obtain a second - level data fusion feature; the second - level data fusion feature is processed through a third - level residual module and fused with the 1 / 8 grayscale image feature map using a third - level feature fusion module to obtain a third - level data fusion feature; the third - level data fusion feature is processed through a fourth - level residual module and fused with the 1 / 16 grayscale image feature map using a fourth - level feature fusion module to obtain a fourth - level data fusion feature; the fourth - level data fusion feature is processed through a fifth - level residual module and added element - by - element to the 1 / 32 grayscale image feature map to obtain a fifth - level data fusion feature, which is used as the multi - modal data fusion feature F t at time t.

[0022] According to the spacecraft depth completion method based on time - series optical images and lidar data of the present invention, the method for obtaining the target depth prediction result at time t in the decoding stage includes:

[0023] Performing temporal relationship modeling on the multi - modal data fusion feature F t at time t through a five - level LSTM module and the five - level feature state at time t - 1 to obtain the five - level temporal enhancement feature at time t and the five - level feature memory state Together with they jointly form the five-level feature state at time t The five-level temporal enhanced feature is decoded through a five-level deconvolution layer to obtain the five-level feature decoding result; the five-level feature decoding result is passed through a four-level LSTM module and the four-level feature state at time t-1 to model the temporal relationship and obtain the four-level temporal enhanced feature at time t and the four-level feature memory state Together with they jointly form the four-level feature state at time t The four-level temporal enhanced feature is decoded through a four-level deconvolution layer to obtain the four-level feature decoding result; the four-level feature decoding result is passed through a three-level LSTM module and the three-level feature state at time t-1 to model the temporal relationship and obtain the three-level temporal enhanced feature at time t and the three-level feature memory state Together with they jointly form the three-level feature state at time t The three-level temporal enhanced feature is decoded through a three-level deconvolution layer to obtain the three-level feature decoding result; the three-level feature decoding result is passed through a two-level LSTM module and the two-level feature state at time t-1 to model the temporal relationship and obtain the two-level temporal enhanced feature at time t and the two-level feature memory state Together with they jointly form the two-level feature state at time t The two-level temporal enhanced feature is decoded through a two-level deconvolution layer to obtain the two-level feature decoding result; the two-level feature decoding result is passed through a one-level LSTM module and the one-level feature state at time t-1 to model the temporal relationship and obtain the one-level temporal enhanced feature at time t and the one-level feature memory state Together with they jointly form the one-level feature state at time t The one-level temporal enhanced feature is decoded through a one-level deconvolution layer to obtain the one-level feature decoding result;

[0024] The one-level feature decoding result is combined with the foreground segmentation image to obtain the target depth prediction result at time t

[0025] According to the spacecraft depth completion method based on temporal optical images and lidar data of the present invention, the foreground segmentation module uses the cascaded image of the sparse depth image I s,t and the target grayscale image I g,t as the input, predicts the probability that each pixel in the cascaded image belongs to the target through an encoder-decoder structure with skip connections, sets the pixel depth prediction results lower than the target threshold to 0, and obtains the image after foreground segmentation.

[0026] According to the spacecraft depth completion method based on temporal optical images and lidar data of the present invention, the simulation conditions of the three-dimensional model of the space target are set to include three-dimensional model material parameter setting, texture mapping setting, light addition, earth background addition, starry sky background addition, relative position and relative attitude setting between the target and the observation platform, and output node setting.

[0027] According to the spacecraft depth completion method based on temporal optical images and lidar data of the present invention, during the training process of the temporal spacecraft depth completion neural network model, the gradient of the network weight parameters is calculated using the error between the target depth prediction result each time and the target dense depth ground truth of the image, and the network parameters are updated.

[0028] According to the spacecraft depth completion method based on temporal optical images and lidar data of the present invention, after the set number of times of network parameter update of the temporal spacecraft depth completion neural network model, the performance of the network model is verified using the sample data of the validation set, and the network parameters corresponding to the optimal performance of the network model are used as the network parameters of the trained temporal spacecraft depth completion neural network model.

[0029] According to the spacecraft depth completion method based on temporal optical images and lidar data of the present invention, for the three-dimensional model of the space target, a three-dimensional rendering software is used for simulation to obtain the temporal sequence of the target grayscale images of the monocular visible light camera and the temporal sequence of the sparse depth images of the lidar.

[0030] According to the spacecraft depth completion method based on temporal optical images and lidar data of the present invention, in the decoding stage, prior knowledge of the spacecraft size is introduced, and the pixel depths with deviations greater than the preset deviation threshold between the target depth prediction result and the average ranging result of the lidar are filtered out.

[0031] Advantages of the present invention: The method of the present invention makes full use of the spatio-temporal coherence contained in consecutive frame data through the LSTM module in the decoding stage, so as to obtain a depth completion result with high precision and consistent time for spatial targets. It first collects the three-dimensional structure model of the spatial target, and uses 3D software to realize the simulation of sensor imaging data and the automatic generation of depth labels under different working conditions and different camera parameters, so as to construct a temporal depth completion dataset for the spatial target; then trains the temporal spacecraft depth completion neural network model and updates the parameters; finally, injects the test data into the trained temporal spacecraft depth completion neural network model, and can realize the completion of the target depth data and the evaluation of the completion accuracy.

[0032] The method of the present invention fully explores the correlation relationship of target information between different frame image data, improves the dense depth accurate recovery of spatial targets, and obtains a temporally stable depth prediction result of spatial targets; it can solve the problem of inconsistent inter-frame completion results caused by existing single-frame depth completion methods ignoring the correlation information of temporal data, and has the advantages of small memory occupancy, high precision, fast speed, consistent temporal prediction results, etc., and realizes the accurate recovery of the three-dimensional fine structure of spatial targets.

[0033] The method of the present invention introduces a recurrent neural network into the standard decoder structure, enabling the network to perceive the changes in target temporal features; the recurrent neural network adopts a multi-level embedding form, enabling the network to adapt to working conditions with different change rates, and the network is more robust. The recurrent neural network can accumulate past target feature knowledge, and the network prediction result becomes more and more accurate over time. Brief Description of the Drawings

[0034] Figure 1 is a schematic flow chart of the spacecraft depth completion method based on temporal optical images and lidar data according to the present invention;

[0035] Figure 2 is the overall framework diagram of the temporal spacecraft depth completion neural network model; the variable with subscript t - 2 in the figure is the corresponding feature state at time t - 2;

[0036] Figure 3 is a schematic flow chart of the feature fusion module in the depth image feature extraction module in the encoding stage;

[0037] Figure 4 is the structure diagram of the long short-term memory network (LSTM module). Detailed Embodiments

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0039] It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments may be combined with each other.

[0040] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but it is not limited to the present invention.

[0041] Specific embodiments: First, in combination with Figures 1 to 3 As shown, the present invention provides a method for spacecraft depth completion based on sequential optical images and lidar data, including

[0042] Collect multiple three-dimensional models of space targets, set the on-orbit imaging simulation conditions of the three-dimensional models of space targets and the sensor parameters of the monocular visible light camera and lidar; based on multiple three-dimensional models of space targets, simulate the sequential gray-scale image sequence of the target of the monocular visible light camera and the sequential sparse depth image sequence of the lidar, and generate the target dense depth ground truth of the images, so as to construct a space target sequential depth completion data set;

[0043] Construct a sequential spacecraft depth completion neural network model and perform network parameter initialization design and hyperparameter setting; the network model includes multiple target depth prediction branches cascaded in chronological order;

[0044] Each target depth prediction branch includes an encoding stage and a decoding stage:

[0045] The encoding stage includes a foreground segmentation module, a gray-scale image feature extraction module, a morphological preprocessing module, and a depth image feature extraction module; the decoding stage includes an LSTM module and a transposed convolutional layer; among them, the depth image feature extraction module includes four feature fusion modules;

[0046] The prediction process of the depth of the space target at time t includes:

[0047] The sparse depth image I at time t s,t After being preprocessed by the morphological preprocessing module, the preprocessed depth image is input into the depth image feature extraction module;

[0048] The target gray-scale image I at time t g,t Extract multi-scale gray-scale image feature maps with different semantic levels and different resolutions through the gray-scale image feature extraction module;

[0049] For the sparse depth image I at time t s,t and the target grayscale image I g,t After performing a cascading operation, it is input into the foreground segmentation module for target foreground segmentation to obtain a foreground segmentation image;

[0050] The depth image feature extraction module first processes the preprocessed depth image through a convolutional layer, and then performs feature fusion with the grayscale image feature map level by level in the order from large to small scale to obtain the multi-modal data fusion feature F at time t t ;

[0051] The multi-modal data fusion feature F at time t t Performs temporal relationship modeling with the corresponding hierarchical feature states at time t - 1 through the LSTM module level by level, and performs feature decoding through a transposed convolutional layer; the feature decoding result of the last layer is combined with the foreground segmentation image to obtain the target depth prediction result at time t;

[0052] Uses the temporal sequence of the target grayscale image of the monocular visible light camera and the temporal sequence of the sparse depth image of the lidar as sample images to train the temporal spacecraft depth completion neural network model, and obtains the trained temporal spacecraft depth completion neural network model;

[0053] Embeds the trained temporal spacecraft depth completion neural network model into the on-board platform, and realizes the prediction of the dense depth of the space target and the restoration of the target three-dimensional structure based on the real-time sensing data collected by the monocular visible light camera and the lidar.

[0054] In this embodiment, the three-dimensional model of the space target can be obtained by collection and purchase.

[0055] The temporal spacecraft depth completion neural network model adopts an encoder-decoder structure as the basic structure of the network to obtain an end-to-end trainable neural network model.

[0056] In the feature encoding of the target grayscale image and the sparse depth image, in order to prevent overly sparse data from damaging the convolutional operation, a morphological preprocessing method is used to obtain pseudo-dense depth data, which is used as the input of the depth image feature extraction module.

[0057] In this embodiment, the LSTM module in the decoding stage is a type of recurrent neural network (RNNs). The LSTM module is hierarchically embedded into the standard decoder structure to capture the feature deviation between frames of different hierarchical feature maps, so as to make full use of the temporal correlation between adjacent frames to generate a temporally stable dense depth map.

[0058] In this embodiment, in order to filter out the interference of irrelevant backgrounds on spacecraft depth completion, a foreground segmentation network in the single-frame spacecraft depth completion network (SDCNet) is used to filter out the starry sky background.

[0059] In this embodiment, in the feature decoding stage, the LSTM module is used to fuse the multi-modal data feature F at time t-1 t with the feature state s at the current time t-1 to model the temporal relationship and output the temporally enhanced feature Temporally enhanced feature Feature decoding is performed through the deconvolution layer.

[0060] Furthermore, as shown in Figure 2 In this embodiment, the target grayscale image I g,t is processed by the grayscale image feature extraction module to extract grayscale image feature maps of five sizes, which are 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the target grayscale image I g,t respectively.

[0061] Further, as shown in Figure 2 The process of the depth image feature extraction module obtaining the multi-modal data fusion feature F at time t includes: t :

[0062] The depth image feature extraction module first processes the preprocessed depth image through the convolutional layer, and then after being processed by the first-level residual module, it is fused with the 1 / 2 grayscale image feature map by the first-level feature fusion module to obtain the first-level data fusion feature; the first-level data fusion feature is processed by the second-level residual module and then fused with the 1 / 4 grayscale image feature map by the second-level feature fusion module to obtain the second-level data fusion feature; the second-level data fusion feature is processed by the third-level residual module and then fused with the 1 / 8 grayscale image feature map by the third-level feature fusion module to obtain the third-level data fusion feature; the third-level data fusion feature is processed by the fourth-level residual module and then fused with the 1 / 16 grayscale image feature map by the fourth-level feature fusion module to obtain the fourth-level data fusion feature; the fourth-level data fusion feature is processed by the fifth-level residual module and then added element-wise to the 1 / 32 grayscale image feature map to obtain the fifth-level data fusion feature, which is used as the multi-modal data fusion feature F at time t t

[0063] Furthermore, as shown in Figure 2 and Figure 3 The method for obtaining the target depth prediction result at time t in the decoding stage includes:

[0064] The multi-modal data fusion feature F at time t t is modeled for the temporal relationship through the five-level LSTM module with the five-level feature state at time t-1 to obtain the five-level temporally enhanced feature at time t and the five-level feature memory state which together with constitute the five-level feature state at time t ​Enhance the five - level temporal features Perform feature decoding through a five - level transposed convolutional layer to obtain the five - level feature decoding result; pass the five - level feature decoding result through a four - level LSTM module and the four - level feature state at time t - 1 Perform temporal relationship modeling to obtain the four - level temporal enhanced feature at time t And the four - level feature memory state And Together form the four - level feature state at time t Perform feature decoding on the four - level temporal enhanced feature through a four - level transposed convolutional layer to obtain the four - level feature decoding result; pass the four - level feature decoding result through a three - level LSTM module and the three - level feature state at time t - 1 Perform temporal relationship modeling to obtain the three - level temporal enhanced feature at time t And the three - level feature memory state And Together form the three - level feature state at time t Perform feature decoding on the three - level temporal enhanced feature through a three - level transposed convolutional layer to obtain the three - level feature decoding result; pass the three - level feature decoding result through a two - level LSTM module and the two - level feature state at time t - 1 Perform temporal relationship modeling to obtain the two - level temporal enhanced feature at time t And the two - level feature memory state And Together form the two - level feature state at time t Perform feature decoding on the two - level temporal enhanced feature through a two - level transposed convolutional layer to obtain the two - level feature decoding result; pass the two - level feature decoding result through a one - level LSTM module and the one - level feature state at time t - 1 Perform temporal relationship modeling to obtain the one - level temporal enhanced feature at time t And the one - level feature memory state And Together form the one - level feature state at time t Perform feature decoding on the one - level temporal enhanced feature through a one - level transposed convolutional layer to obtain the one - level feature decoding result;

[0065] Combine the one - level feature decoding result with the foreground segmentation image to obtain the target depth prediction result at time t

[0066] In this embodiment, the foreground segmentation module uses the sparse depth image I s,t and the target grayscale image I g,tTaking the cascaded image as input, predicting the probability that each pixel in the cascaded image belongs to the target through an encoder-decoder structure with skip connections, setting the pixel depth prediction results lower than the target threshold to 0, and obtaining the foreground-segmented image.

[0067] As an example, the simulation conditions of the spatial target three-dimensional model are set to include three-dimensional model material parameter settings, texture mapping settings, light addition, earth background addition, starry sky background addition, relative position and relative attitude settings between the target and the observation platform, and output node settings.

[0068] Furthermore, during the training process of the time-series spacecraft depth completion neural network model, the gradient of the network weight parameters is calculated using the error between the target depth prediction result and the target dense depth ground truth of the image each time, and the network parameters are updated.

[0069] In this embodiment, after the set number of times of network parameter update of the time-series spacecraft depth completion neural network model, the performance of the network model is verified using the validation set sample data, and the network parameters corresponding to the optimal network model performance are saved as the network parameters of the trained time-series spacecraft depth completion neural network model.

[0070] Taking the saved network parameters as the final network weights, loading the test data and inputting it into the network to obtain the depth completion result, and comparing it with the ground truth for network accuracy evaluation.

[0071] As an example, for the spatial target three-dimensional model, a three-dimensional rendering software is used to simulate the target gray image time series of the monocular visible light camera and the sparse depth image time series of the lidar.

[0072] In this embodiment, in the decoding stage, prior knowledge of the spacecraft size is introduced, and the pixel depths with a deviation greater than the preset deviation threshold between the target depth prediction result and the average ranging result of the lidar are filtered out, so as to obtain a high-quality depth prediction result.

[0073] The working process of the time-series spacecraft depth completion neural network model is described in detail below:

[0074] Combined with Figure 2 As shown, for the target gray image I g,t input at time t and the sparse depth image I s,t , a multi-source depth image feature extraction module is used to aggregate the features of different sensor data, so as to obtain a multi-modal data fusion feature F t , which can be expressed as:

[0075] F t = F encoder ([I g,t , I s,t , θencoder )

[0076] where F encoder is the encoder feature extraction function, and θ encoder represents the network parameters to be learned for the encoder structure.

[0077] Combined with Figure 3 as shown, the feature fusion module in the encoding stage realizes the fusion of the target grayscale image features and the depth image features based on the attention mechanism, providing features with high representational ability for subsequent target depth decoding.

[0078] The feature fusion module mainly consists of a feature embedding layer, a cross-channel fusion layer, and a spatial attention layer; the following is a detailed description of the first-level feature fusion module, and the superscript 1 is added to the first-level features for correspondence:

[0079] The first-level feature fusion module takes the first-level features of the grayscale image and the first-level features of the depth image (C represents the number of feature channels, and H and W represent the length and width of the feature map respectively) as inputs, and obtains the first-level data fusion feature

[0080] The feature embedding layer encodes the feature maps of different channels to generate corresponding feature vectors; the feature embedding layer decomposes the grayscale feature and the depth feature into M non-overlapping regions (the size of each feature block is S×S), and uses depthwise separable convolution operations with a convolution kernel size of S×S and a convolution stride of S to extract the regional features. In addition, for and the regional blocks of, max-pooling operations and average-pooling operations are performed to extract the regional global features, and finally the three features are concatenated to output the first-level embedded feature of the grayscale image and the first-level embedded feature of the depth image where d k = 3×H×W / S 2 .

[0081] The cross-channel attention layer takes the feature encoding results of the feature embedding layer as inputs. First, it uses linear transformations to calculate the query vectors of the first-level embedded features of the grayscale image and the key vectors of the first-level embedded features of the depth image for n attention heads respectively, which can be specifically expressed as:

[0082]

[0083] where are the linear mapping weight matrices for calculating the query vectors and the key vectors respectively, They are the query vector of the grayscale image embedding feature and the key vector of the depth image embedding feature for the $i$-th attention head respectively; $n$ is the number of attention heads.

[0084] Furthermore, the channel correlation weight matrix of the grayscale image feature and the depth image feature for the $i$-th attention head is calculated using scaled dot-product attention, which can be specifically expressed as:

[0085]

[0086] where $w$ i is the feature channel correlation weight for the $i$-th attention head, and softmax(·) is the normalization function.

[0087] Finally, according to the correlation weight matrix, the first-level feature of the grayscale image and the first-level feature of the depth image $F$ s 1 are subjected to channel fusion. It can be specifically expressed as:

[0088]

[0089] where $h$ g and $h$ s are the feature vectors obtained by unfolding the first-level feature of the grayscale image and the first-level feature of the depth image row by row respectively, $w$ i is the feature channel correlation weight for the $i$-th attention head, reshape(·) is the vector dimension transformation operation, is the first-level channel fusion feature for the $i$-th attention head.

[0090] Finally, the channel fusion features calculated by the $n$ attention heads are concatenated, and further convolution operations are used to achieve multi-head attention feature fusion, obtaining the first-level channel fusion feature It can be specifically expressed as:

[0091]

[0092] where Conv(·) is the convolution operation, and [·;·] is the feature concatenation operation.

[0093] The spatial attention layer takes the first-level channel fusion feature and the first-level feature of the grayscale image as inputs, and uses channel average pooling and channel max pooling operations to represent the features at different spatial positions, and concatenates the features and inputs them into the convolutional layer to obtain the first-level spatial attention weight. It can be specifically expressed as:

[0094]

[0095] where is the first-level spatial attention weight; max c (·), avg c (·) are the channel maximum pooling and average pooling operations respectively; σ(·) is the Sigmoid normalization function; Conv(·) and [·;·] are the convolution operation and the feature concatenation operation respectively.

[0096] Based on the first-level spatial attention weight, the features at different spatial positions are weighted and summed to obtain the first-level fusion feature Specifically, it can be expressed as:

[0097]

[0098] In the formula, ⊙ is the matrix element product operation.

[0099] Furthermore, in the feature encoding stage, the four-level fusion feature After being processed by the five-level residual module and added to the elements of the 1 / 32 grayscale image feature map, the five-level data fusion feature is obtained, which is used as the multi-modal data fusion feature F at time t t .

[0100] In the decoding stage, a long short-term memory network (LSTM module) is introduced before each deconvolution operation, as Figure 4 shown. The LSTM module is mainly composed of an input gate i t 、a forget gate f t and an output gate o t to control the information flow of different frames. Taking the five-level LSTM module as an example below, the internal data processing process is described in detail, and the corresponding variables are added with superscript 5 for correspondence:

[0101] The five-level LSTM module takes the memory state of the same-level feature of the previous frame the temporal enhancement feature of the same-level of the previous frame and the current frame fusion feature F t as inputs, and outputs the memory state of the current frame feature and the temporal enhancement feature of the current frame and together constitute the feature state of the current frame The temporal enhancement feature is used as the input for the subsequent deconvolution operation. and The calculation processes of can be specifically expressed as:

[0102]

[0103]

[0104]

[0105]

[0106]

[0107] wherein [·; ·] represents the feature concatenation operation, * represents the convolution operation, ⊙ represents the element-wise multiplication operation, and W f and W i and W o and W c and b f and b i and b o and b c are the parameters to be learned by the long short-term memory network. Figure 4 In Figure 4 , Conv represents the convolution operation, and tanh represents the hyperbolic tangent function.

[0108] In summary, the method of the present invention introduces a recurrent neural network into the decoder structure, enabling the network to mine the target temporal feature changes; adopts a multi-scale mechanism to hierarchically embed the recurrent neural network before different levels of deconvolution operations, enabling the network to adapt to the target feature changes under different motion speeds and different motion modes; finally, by deeply mining the temporal correlation of the temporal data, the network prediction result is more stable, and the depth completion accuracy can be continuously improved over time.

[0109] Although the present invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed, as long as they do not depart from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the different dependent claims and the features described herein can be combined in a manner different from that described in the original claims. It should also be understood that the features described in connection with a single embodiment can be used in other described embodiments.

Claims

1. A method for spacecraft depth completion based on temporal optical images and lidar data, characterized in that including, collecting three-dimensional models of multiple space targets, setting simulation working conditions of the three-dimensional models of space targets, and sensor parameters of a monocular visible light camera and a lidar; simulating multiple three-dimensional models of space targets to obtain a time series of target gray-scale images of the monocular visible light camera and a time series of sparse depth images of the lidar, and generating a target dense depth ground truth label for the images; constructing a time series spacecraft depth completion neural network model: including multiple target depth prediction branches cascaded in chronological order; each target depth prediction branch includes an encoding stage and a decoding stage: the encoding stage includes a foreground segmentation module, a gray-scale image feature extraction module, a morphological preprocessing module, and a depth image feature extraction module; the decoding stage includes an LSTM module and a deconvolution layer; the prediction process of the depth of the space target at time t includes: Sparse depth image I at time t s,t The preprocessed depth image obtained after being preprocessed by the morphological preprocessing module is input to the depth image feature extraction module; The target grayscale image I at time t g,t The multi-scale grayscale image feature maps of different semantic levels and different resolutions are extracted by the grayscale image feature extraction module; For the sparse depth image I at time t s,t and the target grayscale image I g,t After performing a cascading operation, it is input to the foreground segmentation module for target foreground segmentation to obtain a foreground segmentation image; The depth image feature extraction module first processes the preprocessed depth image through a convolutional layer, and then performs feature fusion with the grayscale image feature map level by level in the order of decreasing scale to obtain the multi-modal data fusion feature F at time t t ; Multimodal data fusion feature F at time t t Model the temporal relationship with the corresponding hierarchical feature state at time t-1 level by level through the LSTM module, and perform feature decoding through the transposed convolution layer; the feature decoding result of the last layer is combined with the foreground segmentation image to obtain the target depth prediction result at time t; using the time series of target gray-scale images of the monocular visible light camera and the time series of sparse depth images of the lidar as sample images to train the time series spacecraft depth completion neural network model, and obtaining a trained time series spacecraft depth completion neural network model; embedding the trained time series spacecraft depth completion neural network model into a spaceborne platform, and realizing the prediction of the dense depth of the space target based on the real-time sensing data collected by the monocular visible light camera and the lidar.

2. The method for spacecraft depth completion based on temporal optical images and lidar data according to claim 1, wherein Target grayscale image I g,t The grayscale image feature extraction module extracts grayscale image feature maps of five sizes, and the sizes are 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the target grayscale image I g,t respectively.

3. The spacecraft depth completion method based on time series optical images and lidar data according to claim 2, wherein The deep image feature extraction module obtains the multi-modal data fusion feature F at time t t The process includes: The depth image feature extraction module first processes the preprocessed depth image through a convolutional layer, and then after being processed by a first-level residual module, it is fused with the 1 / 2 grayscale image feature map using a first-level feature fusion module to obtain a first-level data fusion feature; the first-level data fusion feature is processed by a second-level residual module and then fused with the 1 / 4 grayscale image feature map using a second-level feature fusion module to obtain a second-level data fusion feature; the second-level data fusion feature is processed by a third-level residual module and then fused with the 1 / 8 grayscale image feature map using a third-level feature fusion module to obtain a third-level data fusion feature; the third-level data fusion feature is processed by a fourth-level residual module and then fused with the 1 / 16 grayscale image feature map using a fourth-level feature fusion module to obtain a fourth-level data fusion feature; the fourth-level data fusion feature is processed by a fifth-level residual module and then element-wise added to the 1 / 32 grayscale image feature map to obtain a fifth-level data fusion feature, which is used as the multi-modal data fusion feature F at time t t 。 4. The spacecraft depth completion method based on time series optical images and lidar data according to claim 3, wherein the method for obtaining the target depth prediction result at time t in the decoding stage includes: The multi-modal data fusion feature F at time t t Through a five-level LSTM module and the five-level feature state at time t-1 Perform temporal relationship modeling to obtain the five-level temporal enhancement feature at time t And the five-level feature memory state And Together form the five-level feature state at time t The five-level temporal enhancement feature Perform feature decoding through a five-level transposed convolutional layer to obtain the five-level feature decoding result; pass the five-level feature decoding result through a four-level LSTM module and the four-level feature state at time t-1 Perform temporal relationship modeling to obtain the four-level temporal enhancement feature at time t And the four-level feature memory state And Together form the four-level feature state at time t The four-level temporal enhancement feature Perform feature decoding through a four-level transposed convolutional layer to obtain the four-level feature decoding result; pass the four-level feature decoding result through a three-level LSTM module and the three-level feature state at time t-1 Perform temporal relationship modeling to obtain the three-level temporal enhancement feature at time t And the three-level feature memory state And Together form the three-level feature state at time t The three-level temporal enhancement feature Perform feature decoding through a three-level transposed convolutional layer to obtain the three-level feature decoding result; pass the three-level feature decoding result through a two-level LSTM module and the two-level feature state at time t-1 Perform temporal relationship modeling to obtain the two-level temporal enhancement feature at time t And the two-level feature memory state And Together form the two-level feature state at time t The two-level temporal enhancement feature Perform feature decoding through a two-level transposed convolutional layer to obtain the two-level feature decoding result; pass the two-level feature decoding result through a one-level LSTM module and the one-level feature state at time t-1 Perform temporal relationship modeling to obtain the one-level temporal enhancement feature at time t And the one-level feature memory state And Together form the one-level feature state at time t Enhance the first-level timing features Perform feature decoding through the first-level transposed convolutional layer to obtain the first-level feature decoding result; combining the first-level feature decoding result with the foreground segmentation image to obtain the target depth prediction result at time t.

5. The method for spacecraft depth completion based on temporal optical images and lidar data according to claim 4, wherein The foreground segmentation module takes the cascaded image of the sparse depth image I s,t and the target grayscale image I g,t as input, predicts the probability that each pixel in the cascaded image belongs to the target through an encoder-decoder structure with skip connections, sets the pixel depth prediction results below the target threshold to 0, and obtains the image after foreground segmentation.

6. The spacecraft depth completion method based on time series optical images and lidar data according to claim 1, wherein setting the simulation working conditions of the three-dimensional model of the space target includes setting three-dimensional model material parameters, texture mapping, adding light, adding an earth background, adding a starry sky background, setting the relative position and relative attitude between the target and the observation platform, and setting output nodes.

7. The method for spacecraft depth completion based on sequential optical images and lidar data according to claim 1, wherein During the training process of the time series spacecraft depth completion neural network model, the gradient of the network weight parameters is calculated using the error between each target depth prediction result and the target dense depth ground truth label of the image, and the network parameters are updated.

8. The method for spacecraft depth completion based on sequential optical images and lidar data according to claim 7, characterized in that After the set number of times of updating the network parameters of the time series spacecraft depth completion neural network model, the performance of the network model is verified using the sample data of the validation set, and the network parameters corresponding to the optimal performance of the network model are used as the network parameters of the trained time series spacecraft depth completion neural network model.

9. The spacecraft depth completion method based on time series optical images and lidar data according to claim 1, wherein simulating the three-dimensional model of the space target using 3D rendering software to obtain a time series of target gray-scale images of the monocular visible light camera and a time series of sparse depth images of the lidar.

10. The spacecraft depth completion method based on time series optical images and lidar data according to claim 1, wherein In the decoding stage, prior knowledge of the spacecraft size is introduced to filter out the pixel depths whose deviation between the target depth prediction result and the average ranging result of the lidar is greater than the preset deviation threshold.

Images