Spacecraft pose measurement system based on monocular vision image processing

Abstract

The invention discloses a spacecraft pose measurement system based on monocular vision image processing, and belongs to the technical field of spacecraft pose measurement. The invention aims to solve the problems of high equipment cost, high complexity and large measurement error in the conventional spacecraft pose measurement method. Comprising the following steps: obtaining a normalized grey-scale map from a target two-dimensional image; obtaining a target image with a set size by a target detection module; a key point regression module obtains high-dimensional features and two-dimensional key point coordinates of the target by adopting a backbone network based on ResNet; a translation parameter and a rotation parameter of the target are obtained by the multi-layer sensor module, and a pose distribution initial value of the target is obtained; a PnP pose optimization module obtains pose six-degree-of-freedom parameters of the target spacecraft relative to the mother spacecraft; in a measurement system training process, a Monte Carlo method is adopted to construct a loss function based on errors to adjust system network parameters. The method is used for spacecraft pose measurement.

Description

Technical Field

[0001] The invention relates to a spacecraft posture measurement system based on monocular vision image processing, and belongs to the technical field of spacecraft posture measurement. Background Art

[0002] Existing spacecraft posture measurement methods include: stereo vision-based measurement methods, depth camera-based measurement methods, and lidar-based measurement methods.

[0003] Methods based on stereo vision include binocular vision and trinocular vision technology. Stereo vision technology simulates the principle of binocular or multi-eyes to realize the positioning and recognition of objects in three-dimensional space. The position difference of the same object image captured by multiple cameras from different angles, that is, parallax, is used to establish an image solution model to calculate the spacecraft posture. However, the hardware cost and complexity of the method based on stereo vision are high, and the computing resources are consumed a lot; the image solution model is highly dependent on the relative position of multiple cameras and the target, and the limited field of view range depends on the baseline distance of the camera. The imaging range is generally within 6m, and the error is proportional to the square of the object distance. The accuracy is significantly reduced when the target is far away; it is sensitive to lighting conditions, and inconsistent lighting conditions may make it difficult to find the correspondence between multiple images. In addition, the image processing methods used are usually based on manually designed low-order derivative mnemonics such as Harris, SURF, and SIFT, which makes it difficult to collect high-order features of the image. In the existing methods, only by using very complex mnemonics can the average position error of 0.06m and the average attitude error of 2° be achieved on the designed image input.

[0004] The methods based on depth cameras mainly include the time-of-flight method (ToF). Depth cameras can capture three-dimensional information of the scene, not just two-dimensional images. The ToF method continuously sends light pulses to the target, and then uses a sensor to receive the light returned from the object, and obtains the distance of the target by detecting the flight time of the light pulses. The frame rate of the depth camera is relatively high, reaching more than 30Hz. However, the resolution of mainstream ToF sensors is relatively low (for example, 180x240, 240x480, etc.), so the accuracy and X / Y resolution will also be low, and the measurable distance is generally 0.3 to 4.5m; the power consumption and heat generation of ToF components during operation are relatively large, and the complexity of thermal control needs to be enhanced on the spacecraft; at the same time, ToF may also produce multiple reflections of light beams in the concave shape area on the surface of the spacecraft. This phenomenon is called multipath interference, which will cause errors in the calculation of spatial distance by ToF.

[0005] LiDAR-based method: LiDAR can obtain the three-dimensional point cloud data of the target, compare it with the known point cloud model of the spacecraft, perform point cloud data registration, and obtain the optimal posture information as the measurement result. However, the hardware cost and complexity of the LiDAR-based method are the highest among the three methods. The receiver needs to use InGaAs photodetector chips. Although the detection distance is often >10m, the scanning speed of LiDAR is slow, and the data is sparser and more difficult to process than camera images. The common frame rate is 10-20Hz, the real-time performance is low, and the measurement error of high-speed moving objects is large. Summary of the invention

[0006] In view of the problems of high equipment cost and complexity and large measurement error in existing spacecraft posture measurement methods, the present invention provides a spacecraft posture measurement system based on monocular vision image processing.

[0007] A spacecraft posture measurement system based on monocular vision image processing of the present invention comprises:

[0008] A preprocessing module is used to preprocess the target two-dimensional image taken by the monocular camera to obtain a normalized grayscale image;

[0009] The target detection module obtains the region of interest containing the target from the normalized grayscale image and crops it into a target image of a set size;

[0010] The key point regression module uses a ResNet-based backbone network to extract high-dimensional features of target images of a set size and obtain the two-dimensional key point coordinates of the target;

[0011] The multi-layer perceptron module predicts the translation and rotation parameters of the target based on high-dimensional features and two-dimensional key point coordinates, and obtains the initial value of the target's posture distribution;

[0012] PnP pose optimization module: using the correspondence between the coordinates of the two-dimensional key points and the three-dimensional coordinates of the target, the PnP method is used to perform nonlinear optimization on the initial value of the pose distribution to obtain the six-degree-of-freedom parameters of the pose of the target spacecraft relative to the mother spacecraft;

[0013] The monocular camera is arranged on the mother spacecraft platform;

[0014] During the measurement system training process, the Monte Carlo method is used to reproject the target three-dimensional coordinates according to the initial value of the pose distribution. The loss function is constructed based on the error between the two-dimensional coordinates obtained by the reprojection and the true two-dimensional coordinates to adjust the system network parameters.

[0015] According to the spacecraft posture measurement system based on monocular vision image processing of the present invention, the preprocessing module first processes the target two-dimensional image into a three-channel grayscale image, and then performs pixel normalization, scaling the pixel intensity to between 0 and 1 to obtain a normalized grayscale image.

[0016] According to the spacecraft posture measurement system based on monocular vision image processing of the present invention, the key point regression module obtains the two-dimensional key point coordinates of the target through the full convolution upsampling layer and the self-attention layer.

[0017] According to the spacecraft posture measurement system based on monocular vision image processing of the present invention, the operation of the key point regression module on the target image of the set size includes:

[0018] The target images of set sizes are respectively input into three compression excitation modules, and the outputs of the three compression excitation modules are simultaneously input into a four-fold full convolution upsampling module. The outputs of the four-fold full convolution upsampling module are sequentially input into the compression excitation module, the basic convolution module, the two-dimensional maximum pooling module, the basic convolution module, the depthwise separable convolution module and the compression excitation module, and then respectively input into three key point regression branches;

[0019] The modules that operate the input data of each key point regression branch include depthwise separable convolution module, basic convolution module, depthwise separable convolution module, compressed excitation module and two-dimensional convolution module in sequence;

[0020] The first key point regression branch obtains the two-dimensional key point coordinates of the target;

[0021] The high-dimensional feature obtained by the second key point regression branch is the target translation feature vector;

[0022] The high-dimensional feature obtained by the third key point regression branch is the target rotation feature vector;

[0023] When the key point regression module processes data, the modules with the same name passed in sequence are different modules.

[0024] According to the spacecraft posture measurement system based on monocular vision image processing of the present invention, the quadruple full convolution upsampling module includes three upsampling branches, and the functions implemented by each upsampling branch include two-dimensional convolution Conv2d, two-dimensional batch normalization Batchnorm2d and ReLU activation in sequence;

[0025] The input of the two-dimensional convolution Conv2d of the three upsampling branches is the output of a corresponding compression excitation module;

[0026] The ReLU activated output of the first upsampling branch is used as the input of the two-dimensional transposed convolution ConvTranspose2d; the output of the two-dimensional transposed convolution ConvTranspose2d is added to the output of the ReLU activated output of the second upsampling branch, and then the output after two-dimensional batch normalization Batchnorm2d, ReLU activation and two-dimensional transposed convolution ConvTranspose2d operation is added to the output of the ReLU activated output of the third upsampling branch. After two-dimensional batch normalization Batchnorm2d and ReLU activation, the output of the four-fold full convolution upsampling module is obtained as the input of the corresponding compressed excitation module in the key point regression module.

[0027] According to the spacecraft posture measurement system based on monocular vision image processing of the present invention, the operations in the basic convolution module include two-dimensional convolution Conv2d, two-dimensional batch normalization Batchnorm2d and ReLU activation in sequence.

[0028] According to the spacecraft posture measurement system based on monocular vision image processing of the present invention, the operations in the compression excitation module include average pooling Avgpool, linear connection Linear, ReLU activation, linear connection Linear, ReLU activation and Sigmoid activation in sequence.

[0029] According to the spacecraft posture measurement system based on monocular vision image processing of the present invention, the operations in the depth separable convolution module include two-dimensional grouped convolution Conv2d, two-dimensional batch normalization Batchnorm2d, ReLU activation, two-dimensional point-by-point convolution Conv2d, two-dimensional batch normalization Batchnorm2d and ReLU activation in sequence.

[0030] According to the spacecraft posture measurement system based on monocular vision image processing of the present invention, the operation process of the multi-layer perceptron module predicting the translation parameters and rotation parameters of the target from the high-dimensional features and the two-dimensional key point coordinates includes:

[0031] The result of adding the two-dimensional key point coordinates of the target and the target translation feature vector is linearly connected, one-dimensional batch normalized and LeakyReLU activated to obtain result one; linear connection, one-dimensional batch normalization and Sigmoid activation are performed at the same time to obtain result two; the result of multiplying result one and result two is linearly connected, one-dimensional batch normalized, LeakyReLU activated, linearly connected, one-dimensional batch normalized, LeakyReLU activated and linearly connected in sequence to obtain the mean of the Gaussian distribution of the target translation parameter;

[0032] The result of adding the two-dimensional key point coordinates of the target and the target rotation feature vector is linearly connected, one-dimensional batch normalized and LeakyReLU activated to obtain result three; linear connection, one-dimensional batch normalization and Sigmoid activation are performed at the same time to obtain result four; the result of multiplying result three and result four is linearly connected, one-dimensional batch normalized, LeakyReLU activated, linearly connected, one-dimensional batch normalized, LeakyReLU activated and linearly connected in sequence to obtain the mean of the Gaussian distribution of the target rotation parameter;

[0033] The result of adding the two-dimensional key point coordinates of the target and the target translation feature vector is linearly connected, one-dimensional batch normalized and LeakyReLU activated to obtain result one; linear connection, one-dimensional batch normalization and Sigmoid activation are performed at the same time to obtain result two; the result of multiplying result one and result two is linearly connected, one-dimensional batch normalized, LeakyReLU activated, linearly connected, one-dimensional batch normalized, LeakyReLU activated and linearly connected in sequence to obtain the Gaussian distribution covariance of the target translation parameter;

[0034] The result of adding the two-dimensional key point coordinates of the target and the target rotation feature vector is linearly connected, one-dimensional batch normalized and LeakyReLU activated to obtain result three; linear connection, one-dimensional batch normalization and Sigmoid activation are performed at the same time to obtain result four; the result of multiplying result three and result four is linearly connected, one-dimensional batch normalized, LeakyReLU activated, linearly connected, one-dimensional batch normalized, LeakyReLU activated and linearly connected in sequence to obtain the Gaussian distribution covariance of the target rotation parameter;

[0035] The initial value of the target's posture distribution is obtained by the mean of the target's translation parameter Gaussian distribution, the mean of the target's rotation parameter Gaussian distribution, the covariance of the target's translation parameter Gaussian distribution and the covariance of the target's rotation parameter Gaussian distribution.

[0036] According to the spacecraft posture measurement system based on monocular vision image processing of the present invention, the process of nonlinearly optimizing the initial value of the posture distribution includes:

[0037] The two-dimensional coordinates obtained by reprojection are represented as projection2D:

[0038] projection2D=π(r·p3D+t),

[0039] Where π is the reprojection function, r is the initial rotation quaternion vector of the initial value of the pose distribution, p3D is the target three-dimensional coordinate, and t represents the translation parameter;

[0040] The error between the two-dimensional coordinates obtained by reprojection and the true two-dimensional coordinates is expressed as e:

[0041] e=∣∣p2D-projection2D∣∣ 2 ,

[0042] Where p2D represents the real two-dimensional coordinates;

[0043] Based on the error e, r and t are differentiated respectively to generate the Jacobian matrix J;

[0044] Calculate the update step size Δθ based on the error e and the Jacobian matrix J:

[0045] Δθ=(J T J+λI) -1 J T e,

[0046] Where λ is the damping coefficient;

[0047] When the error e meets the set error threshold or the iteration reaches the maximum number of times, the final optimization result is used as the six-degree-of-freedom parameter of the pose;

[0048] During the measurement system training process, the average value E of all reprojection errors e is taken as the loss function:

[0049]

[0050] Where e i is the error corresponding to the i-th reprojection, and n is the number of reprojections;

[0051] E is used as the Monte Carlo loss to measure the automatic differentiation and gradient update of the system.

[0052] Beneficial effects of the present invention: The system of the present invention only needs to use a monocular camera, combined with an image processing method based on computer vision and convolutional neural network, to measure the position and posture of the target spacecraft. It has low hardware cost and complexity, and has no special requirements on the structure, thermal control, installation and other positions of the monocular camera, while meeting high computing speed, real-time performance and high accuracy.

[0053] The method of the present invention does not rely on camera resolution and can process images of any reasonable size. It combines computer vision and convolutional neural networks, including a key point regression network with ResNet as the backbone network, a multi-layer perceptron (MLP) as the probability distribution output layer of the pose, and combines the PnP (Perspective-n-Point) method and the Levenberg-Marquardt method to iteratively optimize the output pose, minimize the reprojection error, and achieve accurate pose measurement. It has the characteristics of a wide range of applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a data processing flow chart of the spacecraft posture measurement system based on monocular vision image processing according to the present invention;

[0055] Figure 2 This is the network architecture diagram of the key point regression module;

[0056] Figure 3 This is the network architecture diagram of the four-fold full convolution upsampling module;

[0057] Figure 4 This is the network architecture diagram of the basic convolution module;

[0058] Figure 5 This is the network architecture diagram of the compression excitation module;

[0059] Figure 6 This is the network architecture diagram of the depth-separable convolution module;

[0060] Figure 7 It is a schematic diagram of the network architecture of the multi-layer perceptron module to obtain the Gaussian distribution mean μt of the target translation parameter and the Gaussian distribution mean μr of the target rotation parameter;

[0061] Figure 8 It is a schematic diagram of the network architecture of the multi-layer perceptron module to obtain the Gaussian distribution covariance ∑t of the target translation parameter and the Gaussian distribution covariance ∑r of the target rotation parameter. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0064] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0065] Specific implementation method 1. Combination Figures 1 to 8 As shown, the present invention provides a spacecraft posture measurement system based on monocular vision image processing, comprising:

[0066] A preprocessing module is used to preprocess the target two-dimensional image taken by the monocular camera to obtain a normalized grayscale image;

[0067] The target detection module uses an object detection network (ODN) to obtain the region of interest (ROI) containing the target from the normalized grayscale image and crop it to a target image of a set size;

[0068] The key point regression module uses a ResNet-based backbone network to extract high-dimensional features from a target image of a set size and obtain the two-dimensional key point coordinates of the target spacecraft in the image.

[0069] The multi-layer perceptron module uses the multi-layer perceptron MLP to predict the translation and rotation parameters of the target based on the high-dimensional features and the two-dimensional key point coordinates, and obtains the initial value of the target's posture distribution; the multi-layer perceptron outputs the probability distribution of the posture;

[0070] PnP pose optimization module: using the correspondence between the coordinates of the two-dimensional key points and the three-dimensional coordinates of the target, the PnP method is used to perform nonlinear optimization on the initial value of the pose distribution, minimize the reprojection error, and obtain the six-degree-of-freedom parameters of the pose of the target spacecraft relative to the mother spacecraft, including three translation parameters and three rotation parameters;

[0071] The monocular camera is arranged on the mother spacecraft platform to photograph the target;

[0072] During the training process of the measurement system, the Monte Carlo method is used to sample a large number of predicted pose parameters and reproject the 3D points. That is, the target three-dimensional coordinates are reprojected according to the initial value of the pose distribution. The loss function is constructed based on the error between the two-dimensional coordinates obtained by the reprojection and the true two-dimensional coordinates to adjust the system network parameters.

[0073] Further, combined with Figure 1 As shown in the figure, the preprocessing module can process the target two-dimensional image of any resolution. First, the target two-dimensional image is processed into a three-channel grayscale image, and then pixel normalization is performed to scale the pixel intensity to between 0 and 1 to obtain a normalized grayscale image. During the image acquisition process, the monocular camera uses appropriate exposure settings to ensure image clarity and reduce the impact of noise. Then the image is normalized to reduce the impact of changes in lighting conditions. Standardization and scaling are performed to adapt to the input requirements of the neural network.

[0074] The object detection module uses the object detection network (ODN) to efficiently identify the target region of interest (ROI) contained in the image, crop and adjust the ROI to the standard size of 224x224 for network input to adapt to image input of any resolution, and ensure that only high-resolution processing of key areas is focused, reducing the amount of calculation while ensuring accuracy.

[0075] The key point regression module uses the residual network with ResNet as the backbone as the feature extractor to extract the high-dimensional features of the spacecraft image layer by layer, extract the features of the target image, and then input the extracted features into the key point regression network. Through the full convolution upsampling layer and self-attention layer, the key points on the spacecraft are finally located to obtain the two-dimensional key point coordinates of the target. Different from traditional methods, this key point detection based on deep learning can learn the high-order features of the target, avoiding the shortcomings of manually designed features, such as Figure 2 shown.

[0076] Combination Figure 2 As shown, the operations of the key point regression module on the target image of the set size include:

[0077] The target images of set sizes are respectively input into three compression excitation modules SE, and the outputs of the three compression excitation modules SE are simultaneously input into the four-fold full convolution upsampling module FCN-4s. The output of the four-fold full convolution upsampling module FCN-4s is sequentially passed through the compression excitation module SE, the basic convolution module Conv-Basic, the two-dimensional maximum pooling module Maxpool2d, the basic convolution module Conv-Basic, the depth-separable convolution module Conv-DW and the compression excitation module SE, and then respectively input into three key point regression branches;

[0078] The modules that operate the input data of each key point regression branch include the depth separable convolution module Conv-DW, the basic convolution module Conv-Basic, the depth separable convolution module Conv-DW, the compression excitation module SE and the two-dimensional convolution module Conv2d.

[0079] The first key point regression branch obtains the two-dimensional key point coordinates of the target;

[0080] The high-dimensional feature obtained by the second key point regression branch is the target translation feature vector;

[0081] The high-dimensional feature obtained by the third key point regression branch is the target rotation feature vector;

[0082] When the key point regression module processes data, the modules with the same name passed in sequence are different modules.

[0083] Combination Figure 3 As shown, the four-fold full convolution upsampling module FCN-4s includes three upsampling branches, and the functions implemented by each upsampling branch include two-dimensional convolution Conv2d, two-dimensional batch normalization Batchnorm2d and ReLU activation in sequence;

[0084] The input of the two-dimensional convolution Conv2d of the three upsampling branches is the output of a corresponding compression excitation module;

[0085] The ReLU activated output of the first upsampling branch is used as the input of the two-dimensional transposed convolution ConvTranspose2d; the output of the two-dimensional transposed convolution ConvTranspose2d is added to the output of the ReLU activated output of the second upsampling branch, and then the output after two-dimensional batch normalization Batchnorm2d, ReLU activation and two-dimensional transposed convolution ConvTranspose2d operation is added to the output of the ReLU activated output of the third upsampling branch. After two-dimensional batch normalization Batchnorm2d and ReLU activation, the output of the four-fold full convolution upsampling module is obtained as the input of the corresponding compressed excitation module in the key point regression module.

[0086] Combination Figure 4 As shown, the operations in the basic convolution module Conv-Basic include two-dimensional convolution Conv2d, two-dimensional batch normalization Batchnorm2d and ReLU activation in sequence.

[0087] Combination Figure 5 As shown, the operations in the compression excitation module SE include average pooling Avgpool, linear connection Linear, ReLU activation, linear connection Linear, ReLU activation and Sigmoid activation in sequence.

[0088] Combination Figure 6 As shown, the operations in the depth-wise separable convolution module Conv-DW include two-dimensional grouped convolution Conv2d (deepwise), two-dimensional batch normalization Batchnorm2d, ReLU activation, two-dimensional point-by-point convolution Conv2d (pointwise), two-dimensional batch normalization Batchnorm2d and ReLU activation.

[0089] The two-dimensional maximum pooling module Maxpool2d performs the following processing on the signal with image size (number of channels, width, height):

[0090] 1) The (3, 224, 224) image passes through the backbone network and outputs three sets of feature maps of different sizes: (128, 28, 28), (256, 14, 14), and (512, 7, 7).

[0091] 2) The feature maps of (128, 28, 28), (256, 14, 14), and (512, 7, 7) are each passed through a compression excitation module to strengthen the network's attention, and the output is still the feature maps of (128, 28, 28), (256, 14, 14), and (512, 7, 7).

[0092] 3) The feature maps of (128, 28, 28), (256, 14, 14), and (512, 7, 7) are upsampled through a 4x full convolution upsampling module, and the (14, 14) feature map and the (7, 7) feature map are upsampled and fused with the (28, 28) feature map to output a (256, 28, 28) feature map.

[0093] 4) (256, 28, 28) further strengthens its attention through a compression excitation module, and then passes through a basic convolution module with a stride of 2 and a maximum pooling module, the size is reduced to 1 / 4, and the number of channels is doubled, (512, 7, 7) feature map, and then passes through a basic convolution module, a depthwise separable convolution module and a compression excitation module to further strengthen the features without changing the size and number of channels, and outputs a (512, 7, 7) feature map.

[0094] The (512, 7, 7) feature map obtained by the two-dimensional maximum pooling module Maxpool2d is sent to three independent regression prediction parts respectively. Each part undergoes depthwise separable convolution-basic convolution-depthwise separable convolution-compressed excitation, and finally uses a two-dimensional convolution module without normalization to directly output the feature map to represent the predicted key point coordinates or feature vectors. The output feature map of the predicted key point coordinates is (2*Num, 1, 1), rearranged into (Num, 2), representing the (x, y) coordinates of Num points, and the output feature map of the predicted translation or rotation feature vector is (Num, 7, 7), rearranged into (Num, 49), representing the deep semantic information of translation or rotation contained in each point.

[0095] Going a step further, combined Figure 7 and Figure 8 As shown in FIG. 1 , the operation process of the multilayer perceptron module to predict the translation parameters and rotation parameters of the target from the high-dimensional features and the two-dimensional key point coordinates includes:

[0096] The result of adding the two-dimensional key point coordinates of the target and the target translation feature vector is linearly connected Linear, one-dimensional batch normalization Batchnorm and LeakyReLU activation to obtain result one; at the same time, linear connection Linear, one-dimensional batch normalization Batchnorm and Sigmoid activation are performed to obtain result two; the result of multiplying result one and result two is linearly connected Linear, one-dimensional batch normalization Batchnorm, LeakyReLU activation, linear connection Linear, one-dimensional batch normalization Batchnorm, LeakyReLU activation and linear connection Linear in turn to obtain the mean of the Gaussian distribution of the target translation parameter;

[0097] The result of adding the two-dimensional key point coordinates of the target and the target rotation feature vector is subjected to linear connection Linear, one-dimensional batch normalization and LeakyReLU activation to obtain result three; at the same time, linear connection Linear, one-dimensional batch normalization Batchnorm and Sigmoid activation are performed to obtain result four; the result of multiplying result three and result four is then subjected to linear connection Linear, one-dimensional batch normalization Batchnorm, LeakyReLU activation, linear connection, one-dimensional batch normalization Batchnorm, LeakyReLU activation and linear connection Linear in turn to obtain the mean of the Gaussian distribution of the target rotation parameter;

[0098] The result of adding the two-dimensional key point coordinates of the target and the target translation feature vector is subjected to linear connection, one-dimensional batch normalization and LeakyReLU activation to obtain result one; linear connection, one-dimensional batch normalization and Sigmoid activation are performed at the same time to obtain result two; the result of multiplying result one and result two is then subjected to linear connection, one-dimensional batch normalization, LeakyReLU activation, linear connection, one-dimensional batch normalization, LeakyReLU activation and linear connection in sequence to obtain the Gaussian distribution covariance of the target translation parameter;

[0099] The result of adding the two-dimensional key point coordinates of the target and the target rotation feature vector is linearly connected Linear, one-dimensional batch normalization Batchnorm and LeakyReLU activation to obtain result three; linear connection Linear, one-dimensional batch normalization Batchnorm and Sigmoid activation are performed at the same time to obtain result four; the result of multiplying result three and result four is linearly connected Linear, one-dimensional batch normalization Batchnorm, LeakyReLU activation, linear connection Linear, one-dimensional batch normalization Batchnorm, LeakyReLU activation and linear connection Linear in turn to obtain the Gaussian distribution covariance of the target rotation parameter;

[0100] The initial value of the target's posture distribution is obtained by the mean of the target's translation parameter Gaussian distribution, the mean of the target's rotation parameter Gaussian distribution, the covariance of the target's translation parameter Gaussian distribution and the covariance of the target's rotation parameter Gaussian distribution.

[0101] The multi-layer perceptron module combines the features extracted from the key point regression network and the predicted two-dimensional coordinates and inputs them into the multi-layer perceptron MLP to predict the probability distribution of the translation and rotation parameters of the target. The MLP further captures the pose information of the spacecraft at different spatial positions, provides a reasonable initial estimate for PnP optimization, and expresses the uncertainty of the translation and rotation parameters with probability distribution, such as Figure 7 and Figure 8 shown.

[0102] The information is processed as follows:

[0103] The key point coordinates of (1) (Num, 2) and the translation or rotation feature vector of (Num, 49) are combined and rearranged into a (51*Num)-dimensional input vector.

[0104] 2) The (51*Num)-dimensional input vector is activated by linear connection + one-dimensional batch normalization + LeakyReLU, and the output is a (2048)-dimensional intermediate vector. The (51*Num)-dimensional input vector is activated by linear connection + one-dimensional batch normalization + Sigmoid, and the output is a (2048)-dimensional intermediate vector.

[0105] 3) Multiply the above two (2048)-dimensional intermediate quantities point by point to transform them into a (2048)-dimensional intermediate vector to enhance attention.

[0106] 4) The obtained (2048)-dimensional intermediate vector is repeatedly activated by linear connection + one-dimensional batch normalization + LeakyReLU twice. The number of channels is changed from (2048) to (1024) and then to (512), gradually reducing the output intermediate vector.

[0107] 5) Finally, the (512)-dimensional intermediate vector is transformed into a Gaussian distribution of translation or rotation through a linear connection. The output mean vector includes the (3)-dimensional three-axis distance Gaussian distribution mean μt representing the translation and the (4)-dimensional quaternion Gaussian distribution mean μr representing the rotation. The output covariance vector includes the (3)-dimensional three-axis distance Gaussian distribution covariance ∑t representing the translation and the (4)-dimensional quaternion Gaussian covariance vr representing the rotation. μt and μr are the preliminary predicted poses t, r, and Σt and Σr are used for subsequent sampling in the probability distribution.

[0108] Going further, PnP pose optimization: Through the known feature points on the target spacecraft 3D model and the corresponding two-dimensional key points detected in the image, the Levenberg-Marquardt method is used to perform nonlinear optimization on the PnP preliminary solution to minimize the reprojection error and improve the accuracy of pose estimation. The process is as follows:

[0109] 1) Given the three-dimensional point coordinates (Num, 3) p3D, the two-dimensional point coordinates (Num, 2) p2D, the initial three-axis distance (3)-dimensional vector t, and the initial quaternion (4)-dimensional vector r, define the objective function, namely the reprojection error, which represents the deviation between the three-dimensional point after being mapped to the image plane through the current solution and the actual two-dimensional point.

[0110] 2) For all 3D points and 2D points, calculate the reprojected position of each point based on the current rotation and translation. Calculate the reprojection error of all points as the difference between the true position of the 2D point and the predicted position.

[0111] Combination Figure 1 As shown in Figure 1, the specific process of nonlinear optimization of the initial value of the posture distribution includes:

[0112] The two-dimensional coordinates obtained by reprojection are represented as projection2D:

[0113] projection2D=π(r·p3D+t),

[0114] Where π is the reprojection function, r is the initial rotation quaternion vector of the initial value of the pose distribution, p3D is the target three-dimensional coordinate, and t represents the translation parameter;

[0115] The error between the two-dimensional coordinates obtained by reprojection and the true two-dimensional coordinates is expressed as e:

[0116] e=∣∣p2D-projection2D∣∣ 2 ,

[0117] Where p2D represents the real two-dimensional coordinates;

[0118] Based on the error e, r and t are differentiated respectively to generate the Jacobian matrix J;

[0119] The update step size Δθ is calculated based on the error e and the Jacobian matrix J. The rotation quaternion (r+Δr) and translation vector (t+Δt) are updated using the Levenberg-Marquardt update formula, and the quaternion is normalized:

[0120] Δθ=(J T J+λI) -1 J T e,

[0121] Where λ is the damping coefficient;

[0122] Convergence judgment, check the change in error, or whether the maximum number of iterations has been reached. When the error e meets the set error threshold or the maximum number of iterations has been reached, stop the optimization and use the final optimization result as the six-degree-of-freedom parameter of the pose, including the updated quaternion r opt and the translation vector t opt ;

[0123] Reprojection error correction and optimization: Use the reprojection error as the loss function, use the Monte Carlo method to sample the probability distribution of the predicted pose in large quantities, reproject the 3D points, compare the calculated errors with the key points in the actual image, optimize the network parameters, and improve the accuracy and robustness of the pose estimation. The process is as follows:

[0124] 1) Randomly sample a set of translations t1, t2, ..., t from the translation Gaussian distribution N(μt, Σt) n , randomly sample a set of quaternions r1, r2, ..., r from the rotation Gaussian distribution N(μr, Σr) n , each sampling constitutes a complete pose (r1, t1), (r2, t2), ..., (r n , t n ), the sampling number n is 100 to 1000 which is a more reasonable value.

[0125] 2) For each set of sampled poses (r, t) i , calculate the reprojection error of all points.

[0126] projection2D i =π(r i p3D+t i ),

[0127] e i =||p2D-projection2D i ∣∣ 2 ;

[0128] During the measurement system training process, the average value E of all reprojection errors e is taken as the loss function:

[0129]

[0130] Where e i is the error corresponding to the i-th reprojection, and n is the number of reprojections;

[0131] E is used as the Monte Carlo loss to measure the automatic differentiation and gradient update of the system.

[0132] The experimental results show that the system of the present invention has the following advantages:

[0133] The key point extraction step in the system of the present invention is improved: compared with the traditional key point extraction based on Harris, SIFT, SURF and other algorithms, the system of the present invention designs a key point regression network for feature extraction and key point regression, which can capture the complex texture and structural features of the spacecraft surface, especially when the lighting conditions change and the target surface has high-order details.

[0134] Regarding real-time performance and computational complexity: The system of the present invention has a high computing speed due to the use of a lightweight network structure and a method for quickly solving PnP. Only 9GFLOPS are required for one inference, which can meet the real-time requirements. Compared with the calculation of binocular vision and the point cloud registration of the lidar method, the data processing volume of this embodiment is relatively small, which is suitable for the limited computing resources on the spacecraft. In order to further improve the real-time performance of the model, the network parameter quantization method can also be used to reduce the computational overhead of the model, thereby improving the execution efficiency on the embedded spacecraft platform.

[0135] Advantages of monocular vision: This implementation significantly reduces hardware costs and system complexity by using monocular vision. Unlike stereo vision, monocular cameras do not require strict camera calibration and baseline length control, and have a higher degree of freedom in installation position, making them suitable for complex installation environments of spacecraft.

[0136] Precision and robustness: Compared with ToF cameras, monocular cameras have higher planar resolution and can more accurately extract key feature points on spacecraft. The accuracy of pose estimation is further improved by combining the optimization of reprojection errors. If multiple lighting conditions are simulated during deep learning network training, this implementation can still stably extract feature points in scenes where lighting conditions change significantly. In addition, combining the RANSAC algorithm to screen the detected key points can effectively suppress the influence of noise points and improve the robustness of pose estimation.

[0137] Application scenarios of the system of the present invention:

[0138] 1. Autonomous rendezvous and docking of spacecraft: The system of the present invention is suitable for autonomous rendezvous and docking tasks between spacecraft. The monocular camera can obtain the position and posture of the target spacecraft in real time, and guide the mother spacecraft to make precise relative displacement and attitude adjustment. In the early stage of rendezvous, the monocular camera can be used for long-distance target detection and preliminary position and posture estimation, providing basic data for further rendezvous and approach.

[0139] 2. Space station target monitoring: The system of the present invention is suitable for monitoring target spacecraft in the space station. Through real-time posture measurement, the safety of the space station's docking target is ensured. The monocular vision system continuously obtains the posture changes of the target spacecraft, providing early warning and target status monitoring for the space station.

[0140] 3. On-orbit service and maintenance: The system of the present invention is suitable for on-orbit service tasks based on monocular vision key point detection and posture estimation, providing high-precision target position and posture data for maintenance tools such as robotic arms, ensuring the accuracy and safety of maintenance operations.

[0141] The end-to-end pose output design based on probability distribution in the system of the present invention is as follows: the key point features are processed by a multi-layer perceptron, and the probability distribution of the pose of the target spacecraft is output, which takes into account the uncertainty distribution of the pose and improves the description of the uncertainty of the pose.

[0142] Key point regression network design: Key point regression is performed on the features output by the residual network to improve the accuracy of spacecraft feature extraction, especially in complex lighting conditions and high-order detail features.

[0143] Loss function designed by combining probability distribution and Monte Carlo method: Based on the probability distribution of pose prediction, multiple samplings are performed in combination with the Monte Carlo method to construct a more appropriate loss function, thereby improving the robustness and accuracy of the model to the uncertainty of pose estimation.

[0144] The system of the present invention adopts an end-to-end direct method in posture detection: high-order features are extracted through convolutional neural networks, and the accuracy of posture estimation is improved by combining reprojection error optimization. The probability distribution of posture is directly output as the measurement result, and then the Levenberg-Marquardt method is used for nonlinear optimization, avoiding the proxy loss caused by only regressing two-dimensional coordinate points to calculate PnP. It has been verified that on the ESA monocular vision spacecraft image dataset SPEEDPLUS, only a small amount of training is required to achieve an average distance error of 0.085m and an average angle error of 3.4°.

[0145] Low computing resource consumption and strong real-time performance: Compared with lidar or binocular cameras, monocular cameras have a smaller amount of image processing data, which is suitable for the limited computing power of embedded platforms. One inference only requires 9GFLOPS. After data quantization, the theoretical speed can exceed 100Hz on a 1TOPS NPU, which is much higher than lidar and can process spacecraft pose estimation in a more real-time manner.

[0146] The image input resolution is large and the observation distance is far: Compared with depth cameras or stereo vision, which are limited by hardware and have a short observation distance, monocular cameras can achieve an observation distance of 0.5 to 50 meters depending on the lens. The resolution depends on the sensing capability of the camera sensor and can range from millions to 20 million pixels.

[0147] Low hardware cost and complexity: Only a monocular camera is needed, without the need for complex sensor designs such as lidar or stereo vision, thus reducing the system hardware cost and structural complexity.

[0148] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that the features described in conjunction with a single embodiment may be used in other described embodiments.

Claims

1. A spacecraft posture measurement system based on monocular vision image processing, characterized in that: include: A preprocessing module is used to preprocess the target two-dimensional image taken by the monocular camera to obtain a normalized grayscale image; The target detection module obtains the region of interest containing the target from the normalized grayscale image and crops it into a target image of a set size; The key point regression module uses a ResNet-based backbone network to extract high-dimensional features of target images of a set size and obtain the two-dimensional key point coordinates of the target; The multi-layer perceptron module predicts the translation and rotation parameters of the target based on high-dimensional features and two-dimensional key point coordinates, and obtains the initial value of the target's posture distribution; PnP pose optimization module: using the correspondence between the coordinates of the two-dimensional key points and the three-dimensional coordinates of the target, the PnP method is used to perform nonlinear optimization on the initial value of the pose distribution to obtain the six-degree-of-freedom parameters of the pose of the target spacecraft relative to the mother spacecraft; The monocular camera is arranged on the mother spacecraft platform; During the measurement system training process, the Monte Carlo method is used to reproject the target three-dimensional coordinates according to the initial value of the pose distribution. The loss function is constructed based on the error between the two-dimensional coordinates obtained by the reprojection and the true two-dimensional coordinates to adjust the system network parameters.

2. The spacecraft posture measurement system based on monocular vision image processing according to claim 1, characterized in that: The preprocessing module first processes the target two-dimensional image into a three-channel grayscale image, and then performs pixel normalization to scale the pixel intensity to between 0 and 1 to obtain a normalized grayscale image.

3. The spacecraft posture measurement system based on monocular vision image processing according to claim 2, characterized in that: The key point regression module obtains the two-dimensional key point coordinates of the target through the fully convolutional upsampling layer and the self-attention layer.

4. The spacecraft posture measurement system based on monocular vision image processing according to claim 3 is characterized in that: The operations of the key point regression module on the target image of the set size include: The target images of set sizes are respectively input into three compression excitation modules, and the outputs of the three compression excitation modules are simultaneously input into a four-fold full convolution upsampling module. The outputs of the four-fold full convolution upsampling module are sequentially input into the compression excitation module, the basic convolution module, the two-dimensional maximum pooling module, the basic convolution module, the depthwise separable convolution module and the compression excitation module, and then respectively input into three key point regression branches; The modules that operate the input data of each key point regression branch include depthwise separable convolution module, basic convolution module, depthwise separable convolution module, compressed excitation module and two-dimensional convolution module in sequence; The first key point regression branch obtains the two-dimensional key point coordinates of the target; The high-dimensional feature obtained by the second key point regression branch is the target translation feature vector; The high-dimensional feature obtained by the third key point regression branch is the target rotation feature vector; When the key point regression module processes data, the modules with the same name passed in sequence are different modules.

5. The spacecraft posture measurement system based on monocular vision image processing according to claim 4, characterized in that: The four-fold full convolution upsampling module includes three upsampling branches, and the functions implemented by each upsampling branch include two-dimensional convolution Conv2d, two-dimensional batch normalization Batchnorm2d and ReLU activation; The input of the two-dimensional convolution Conv2d of the three upsampling branches is the output of a corresponding compression excitation module; The ReLU activated output of the first upsampling branch is used as the input of the two-dimensional transposed convolution ConvTranspose2d; the output of the two-dimensional transposed convolution ConvTranspose2d is added to the output of the ReLU activated output of the second upsampling branch, and then the output after two-dimensional batch normalization Batchnorm2d, ReLU activation and two-dimensional transposed convolution ConvTranspose2d operation is added to the output of the ReLU activated output of the third upsampling branch. After two-dimensional batch normalization Batchnorm2d and ReLU activation, the output of the four-fold full convolution upsampling module is obtained as the input of the corresponding compressed excitation module in the key point regression module.

6. The spacecraft posture measurement system based on monocular vision image processing according to claim 5, characterized in that: The operations in the basic convolution module include two-dimensional convolution Conv2d, two-dimensional batch normalization Batchnorm2d and ReLU activation in sequence.

7. The spacecraft posture measurement system based on monocular vision image processing according to claim 6, characterized in that: The operations in the compression excitation module include average pooling Avgpool, linear connection Linear, ReLU activation, linear connection Linear, ReLU activation and Sigmoid activation.

8. The spacecraft posture measurement system based on monocular vision image processing according to claim 7, characterized in that: The operations in the depthwise separable convolution module include 2D grouped convolution Conv2d, 2D batch normalization Batchnorm2d, ReLU activation, 2D pointwise convolution Conv2d, 2D batch normalization Batchnorm2d, and ReLU activation in sequence.

9. The spacecraft posture measurement system based on monocular vision image processing according to claim 8, characterized in that: The operation process of the multi-layer perceptron module to predict the translation parameters and rotation parameters of the target from high-dimensional features and two-dimensional key point coordinates includes: The result of adding the two-dimensional key point coordinates of the target and the target translation feature vector is linearly connected, one-dimensional batch normalized and LeakyReLU activated to obtain result one; linear connection, one-dimensional batch normalization and Sigmoid activation are performed at the same time to obtain result two; the result of multiplying result one and result two is linearly connected, one-dimensional batch normalized, LeakyReLU activated, linearly connected, one-dimensional batch normalized, LeakyReLU activated and linearly connected in sequence to obtain the mean of the Gaussian distribution of the target translation parameter; The result of adding the two-dimensional key point coordinates of the target and the target rotation feature vector is linearly connected, one-dimensional batch normalized and LeakyReLU activated to obtain result three; linear connection, one-dimensional batch normalization and Sigmoid activation are performed at the same time to obtain result four; the result of multiplying result three and result four is linearly connected, one-dimensional batch normalized, LeakyReLU activated, linearly connected, one-dimensional batch normalized, LeakyReLU activated and linearly connected in sequence to obtain the mean of the Gaussian distribution of the target rotation parameter; The result of adding the two-dimensional key point coordinates of the target and the target translation feature vector is linearly connected, one-dimensional batch normalized and LeakyReLU activated to obtain result one; linear connection, one-dimensional batch normalization and Sigmoid activation are performed at the same time to obtain result two; the result of multiplying result one and result two is linearly connected, one-dimensional batch normalized, LeakyReLU activated, linearly connected, one-dimensional batch normalized, LeakyReLU activated and linearly connected in sequence to obtain the Gaussian distribution covariance of the target translation parameter; The result of adding the two-dimensional key point coordinates of the target and the target rotation feature vector is linearly connected, one-dimensional batch normalized and LeakyReLU activated to obtain result three; linear connection, one-dimensional batch normalization and Sigmoid activation are performed at the same time to obtain result four; the result of multiplying result three and result four is linearly connected, one-dimensional batch normalized, LeakyReLU activated, linearly connected, one-dimensional batch normalized, LeakyReLU activated and linearly connected in sequence to obtain the Gaussian distribution covariance of the target rotation parameter; The initial value of the target's posture distribution is obtained by the mean of the target's translation parameter Gaussian distribution, the mean of the target's rotation parameter Gaussian distribution, the covariance of the target's translation parameter Gaussian distribution and the covariance of the target's rotation parameter Gaussian distribution.

10. The spacecraft posture measurement system based on monocular vision image processing according to claim 9, characterized in that: The process of nonlinear optimization of the initial value of the pose distribution includes: The two-dimensional coordinates obtained by reprojection are represented as projection2D: projection2D=π(r·p3D+t), Where π is the reprojection function, r is the initial rotation quaternion vector of the initial value of the pose distribution, p3D is the target three-dimensional coordinate, and t represents the translation parameter; The error between the two-dimensional coordinates obtained by reprojection and the true two-dimensional coordinates is expressed as e: e=∣∣p2D-projection2D∣∣ 2 , Where p2D represents the real two-dimensional coordinates; Based on the error e, r and t are differentiated respectively to generate the Jacobian matrix J; Calculate the update step size Δθ based on the error e and the Jacobian matrix J: Δθ=(J T J+λI) -1 J T e, Where λ is the damping coefficient; When the error e meets the set error threshold or the iteration reaches the maximum number of times, the final optimization result is used as the six-degree-of-freedom parameter of the pose; During the measurement system training process, the average value E of all reprojection errors e is taken as the loss function: Where e i is the error corresponding to the i-th reprojection, and n is the number of reprojections; E is used as the Monte Carlo loss to measure the automatic differentiation and gradient update of the system.

Images