Spacecraft posture measurement system based on monocular vision image processing

Abstract

A spacecraft posture measurement system based on monocular vision image processing belongs to the technical field of spacecraft posture measurement. The present invention addresses the problems of high equipment cost and complexity, and large measurement errors in existing spacecraft posture measurement methods. It comprises: obtaining a normalized grayscale image from a two-dimensional target image; obtaining a target image of a set size by a target detection module; obtaining high-dimensional features and two-dimensional key point coordinates of the target by a key point regression module using a backbone network based on ResNet; obtaining the translation parameters and rotation parameters of the target and the initial value of the posture distribution of the target by a multi-layer perceptron module; obtaining the six-degree-of-freedom parameters of the posture of the target spacecraft relative to the mother spacecraft by a PnP posture optimization module; and adjusting the system network parameters by using the Monte Carlo method to construct a loss function based on the error during the measurement system training process. The present invention is used for spacecraft posture measurement.

Description

Technical Field

[0001] The invention relates to a spacecraft posture measurement system based on monocular vision image processing, and belongs to the technical field of spacecraft posture measurement. Background Art

[0002] Existing spacecraft posture measurement methods include: stereo vision-based measurement methods, depth camera-based measurement methods, and lidar-based measurement methods.

[0003] Stereo vision-based methods include binocular and trinocular vision. Stereo vision simulates the principles of binocular or multi-eye vision to locate and identify objects in three-dimensional space. The positional differences (parallax) between images of the same object captured from different angles by multiple cameras are used to build an image solution model, which in turn calculates the spacecraft's pose. However, stereo vision-based methods are hardware-costly and complex, consuming significant computational resources. The image solution model is highly dependent on the relative positions of the multiple cameras and the target, with a limited field of view that depends on the camera's baseline distance. The imaging range is generally within 6 meters, and the error is proportional to the square of the object distance, with accuracy significantly decreasing at greater distances. Stereo vision is also sensitive to lighting conditions, as inconsistent lighting conditions can make it difficult to find correspondences between multiple images. Furthermore, the image processing methods used are often based on handcrafted low-order derivative mnemonics such as Harris, SURF, and SIFT, which struggle to capture higher-order image features. Existing methods use highly complex mnemonics to achieve average position errors of 0.06 meters and average pose errors of 2 degrees on designed image inputs.

[0004] Methods based on depth cameras primarily include time-of-flight (ToF). Depth cameras can capture three-dimensional information about a scene, not just a two-dimensional image. ToF works by continuously sending light pulses to a target, then using a sensor to receive the light returning from the object. The distance to the target is determined by detecting the time of flight of the light pulses. Depth cameras have a high frame rate, exceeding 30Hz. However, mainstream ToF sensors have relatively low resolutions (e.g., 180x240, 240x480, etc.), resulting in lower accuracy and X / Y resolution, and the measurable distance is generally between 0.3 and 4.5 meters. ToF components consume relatively high power and generate relatively high heat during operation, necessitating increased thermal control complexity on spacecraft. Furthermore, ToF sensors can generate multiple reflections of the light beam in concave areas on the spacecraft surface. This phenomenon, known as multipath interference, can cause errors in ToF calculations of spatial distance.

[0005] LiDAR-based methods: LiDAR can acquire 3D point cloud data of the target, compare it with the known point cloud model of the spacecraft, perform point cloud data registration, and obtain the optimal pose information as the measurement result. However, the hardware cost and complexity of this method are the highest among the three methods. The receiver requires an InGaAs photodetector chip. Although the detection range is often greater than 10 meters, LiDAR has a slow scanning speed, and the data is sparser and more difficult to process than camera images. The common frame rate is 10-20Hz, which has low real-time performance and large measurement errors for high-speed moving objects. Summary of the Invention

[0006] In view of the problems of high equipment cost and complexity and large measurement error in existing spacecraft posture measurement methods, the present invention provides a spacecraft posture measurement system based on monocular vision image processing.

[0007] A spacecraft posture measurement system based on monocular vision image processing of the present invention comprises:

[0008] The preprocessing module is used to preprocess the target two-dimensional image captured by the monocular camera to obtain a normalized grayscale image;

[0009] The target detection module obtains the region of interest containing the target from the normalized grayscale image and crops it into a target image of a set size;

[0010] The key point regression module uses a ResNet-based backbone network to extract high-dimensional features from target images of a set size and obtain the two-dimensional key point coordinates of the target;

[0011] The multi-layer perceptron module predicts the translation and rotation parameters of the target based on high-dimensional features and two-dimensional key point coordinates, and obtains the initial value of the target's pose distribution;

[0012] PnP pose optimization module: Using the correspondence between the coordinates of the two-dimensional key points and the three-dimensional coordinates of the target, the PnP method is used to perform nonlinear optimization on the initial value of the pose distribution to obtain the six-degree-of-freedom parameters of the pose of the target spacecraft relative to the mother spacecraft;

[0013] The monocular camera is arranged on the mother spacecraft platform;

[0014] During the measurement system training process, the Monte Carlo method is used to reproject the target three-dimensional coordinates according to the initial value of the pose distribution. The loss function is constructed based on the error between the two-dimensional coordinates obtained by the reprojection and the true two-dimensional coordinates to adjust the system network parameters.

[0015] According to the spacecraft posture measurement system based on monocular vision image processing of the present invention, the preprocessing module first processes the target two-dimensional image into a three-channel grayscale image, and then performs pixel normalization, scaling the pixel intensity to between 0 and 1 to obtain a normalized grayscale image.

[0016] According to the spacecraft pose measurement system based on monocular vision image processing of the present invention, the key point regression module obtains the two-dimensional key point coordinates of the target through the full convolution upsampling layer and the self-attention layer.

[0017] According to the spacecraft pose measurement system based on monocular vision image processing of the present invention, the operations of the key point regression module on the target image of the set size include:

[0018] The target image of a set size is input into three compression excitation modules respectively, and the outputs of the three compression excitation modules are simultaneously input into the four-fold full convolution upsampling module. The output of the four-fold full convolution upsampling module is sequentially passed through the compression excitation module, the basic convolution module, the two-dimensional maximum pooling module, the basic convolution module, the depth-separable convolution module and the compression excitation module, and then respectively input into three key point regression branches;

[0019] The modules that operate on the input data of each key point regression branch include depth-wise separable convolution module, basic convolution module, depth-wise separable convolution module, compressed excitation module and two-dimensional convolution module in sequence;

[0020] The first key point regression branch obtains the two-dimensional key point coordinates of the target;

[0021] The high-dimensional feature obtained by the second key point regression branch is the target translation feature vector;

[0022] The high-dimensional feature obtained by the third key point regression branch is the target rotation feature vector;

[0023] When the key point regression module processes data, the modules with the same name that are passed through in sequence are different modules.

[0024] According to the spacecraft pose measurement system based on monocular vision image processing of the present invention, the quadruple full convolution upsampling module includes three upsampling branches, and the functions implemented by each upsampling branch include two-dimensional convolution Conv2d, two-dimensional batch normalization Batchnorm2d and ReLU activation in sequence;

[0025] The input of the two-dimensional convolution Conv2d of the three upsampling branches is the output of a corresponding compression excitation module;

[0026] The ReLU activated output of the first upsampling branch is used as the input of the two-dimensional transposed convolution ConvTranspose2d; the output of the two-dimensional transposed convolution ConvTranspose2d is added to the output of the ReLU activated output of the second upsampling branch, and then the output after the two-dimensional batch normalization Batchnorm2d, ReLU activation and two-dimensional transposed convolution ConvTranspose2d operation is added to the output of the ReLU activated output of the third upsampling branch. After two-dimensional batch normalization Batchnorm2d and ReLU activation, the output of the four-fold full convolution upsampling module is obtained as the input of the corresponding compressed excitation module in the key point regression module.

[0027] According to the spacecraft pose measurement system based on monocular vision image processing of the present invention, the operations in the basic convolution module include two-dimensional convolution Conv2d, two-dimensional batch normalization Batchnorm2d and ReLU activation in sequence.

[0028] According to the spacecraft pose measurement system based on monocular vision image processing of the present invention, the operations in the compression excitation module include average pooling Avgpool, linear connection Linear, ReLU activation, linear connection Linear, ReLU activation and Sigmoid activation in sequence.

[0029] According to the spacecraft pose measurement system based on monocular vision image processing of the present invention, the operations in the depthwise separable convolution module include two-dimensional grouped convolution Conv2d, two-dimensional batch normalization Batchnorm2d, ReLU activation, two-dimensional point-by-point convolution Conv2d, two-dimensional batch normalization Batchnorm2d and ReLU activation in sequence.

[0030] According to the spacecraft pose measurement system based on monocular vision image processing of the present invention, the operation process of the multi-layer perceptron module to predict the translation parameters and rotation parameters of the target from the high-dimensional features and the two-dimensional key point coordinates includes:

[0031] The result of adding the two-dimensional key point coordinates of the target and the target translation feature vector is linearly connected, one-dimensional batch normalized and LeakyReLU activated to obtain result one; linear connection, one-dimensional batch normalization and Sigmoid activation are performed at the same time to obtain result two; the result of multiplying result one and result two is linearly connected, one-dimensional batch normalization, LeakyReLU activation, linear connection, one-dimensional batch normalization, LeakyReLU activation and linear connection in sequence to obtain the mean of the Gaussian distribution of the target translation parameter;

[0032] The result of adding the two-dimensional key point coordinates of the target and the target rotation feature vector is linearly connected, one-dimensional batch normalized and LeakyReLU activated to obtain result three; linear connection, one-dimensional batch normalization and Sigmoid activation are performed at the same time to obtain result four; the result of multiplying result three and result four is linearly connected, one-dimensional batch normalization, LeakyReLU activation, linear connection, one-dimensional batch normalization, LeakyReLU activation and linear connection in sequence to obtain the mean of the Gaussian distribution of the target rotation parameter;

[0033] The result of adding the two-dimensional key point coordinates of the target and the target translation feature vector is linearly connected, one-dimensional batch normalized and LeakyReLU activated to obtain result one; linear connection, one-dimensional batch normalization and Sigmoid activation are performed at the same time to obtain result two; the result of multiplying result one and result two is linearly connected, one-dimensional batch normalization, LeakyReLU activation, linear connection, one-dimensional batch normalization, LeakyReLU activation and linear connection in sequence to obtain the Gaussian distribution covariance of the target translation parameter;

[0034] The result of adding the two-dimensional key point coordinates of the target and the target rotation feature vector is linearly connected, one-dimensional batch normalized and LeakyReLU activated to obtain result three; linear connection, one-dimensional batch normalization and Sigmoid activation are performed at the same time to obtain result four; the result of multiplying result three and result four is linearly connected, one-dimensional batch normalization, LeakyReLU activation, linear connection, one-dimensional batch normalization, LeakyReLU activation and linear connection in sequence to obtain the Gaussian distribution covariance of the target rotation parameter;

[0035] The initial value of the target's pose distribution is obtained by the mean of the target's translation parameter Gaussian distribution, the mean of the target's rotation parameter Gaussian distribution, the covariance of the target's translation parameter Gaussian distribution, and the covariance of the target's rotation parameter Gaussian distribution.

[0036] According to the spacecraft posture measurement system based on monocular vision image processing of the present invention, the process of performing nonlinear optimization on the initial value of the posture distribution includes:

[0037] The two-dimensional coordinates obtained by reprojection are represented as projection2D:

[0038] projection2D=π(r·p3D+t),

[0039] Where π is the reprojection function, r is the initial rotation quaternion vector of the initial value of the pose distribution, p3D is the target three-dimensional coordinate, and t represents the translation parameter;

[0040] The error between the two-dimensional coordinates obtained by reprojection and the true two-dimensional coordinates is expressed as e:

[0041] e=∣∣p2D-projection2D∣∣ 2 ,

[0042] Where p2D represents the real two-dimensional coordinate;

[0043] Based on the error e, we differentiate r and t to generate the Jacobian matrix J.

[0044] Calculate the update step size Δθ based on the error e and the Jacobian matrix J:

[0045] Δθ=(J T J+λI) -1 J T e,

[0046] Where λ is the damping coefficient;

[0047] When the error e meets the set error threshold or the iteration reaches the maximum number of times, the final optimization result is used as the six-degree-of-freedom parameter of the pose;

[0048] During the measurement system training process, the average E of all reprojection errors e is taken as the loss function:

[0049]

[0050] Where e i is the error corresponding to the i-th reprojection, and n is the number of reprojections;

[0051] E is used as the Monte Carlo loss to measure the automatic differentiation and gradient update of the system.

[0052] The present invention provides the following benefits: The system utilizes only a single monocular camera, combined with image processing methods based on computer vision and convolutional neural networks, to measure the position and orientation of target spacecraft. This system offers low hardware cost and complexity, with no specific requirements for the monocular camera's structure, thermal control, or mounting location. It also achieves high computational speed, real-time performance, and high accuracy.

[0053] The proposed method is independent of camera resolution and can process images of any reasonable size. It combines computer vision with convolutional neural networks, including a keypoint regression network with a ResNet backbone and a multilayer perceptron (MLP) as the output layer for the pose probability distribution. It also incorporates the PnP (Perspective-n-Point) method and the Levenberg-Marquardt method to iteratively optimize the output pose, minimizing reprojection error and achieving accurate pose measurement. This method has a wide range of applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a data processing flow chart of the spacecraft posture measurement system based on monocular vision image processing according to the present invention;

[0055] Figure 2 This is the network architecture diagram of the key point regression module;

[0056] Figure 3 This is the network architecture diagram of the four-fold full convolution upsampling module;

[0057] Figure 4 This is the network architecture diagram of the basic convolution module;

[0058] Figure 5 This is the network architecture diagram of the compression excitation module;

[0059] Figure 6 This is the network architecture diagram of the depth-separable convolution module;

[0060] Figure 7 This is a schematic diagram of the network architecture of the multi-layer perceptron module to obtain the Gaussian distribution mean μt of the target translation parameter and the Gaussian distribution mean μr of the target rotation parameter;

[0061] Figure 8 It is a schematic diagram of the network architecture of the multi-layer perceptron module to obtain the Gaussian distribution covariance ∑t of the target translation parameter and the Gaussian distribution covariance ∑r of the target rotation parameter. DETAILED DESCRIPTION

[0062] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0063] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0064] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0065] Specific implementation method 1. Combination Figures 1 to 8 As shown, the present invention provides a spacecraft posture measurement system based on monocular vision image processing, comprising:

[0066] The preprocessing module is used to preprocess the target two-dimensional image captured by the monocular camera to obtain a normalized grayscale image;

[0067] The target detection module uses an object detection network (ODN) to obtain the region of interest (ROI) containing the target from the normalized grayscale image and crop it to a target image of a set size.

[0068] The key point regression module uses a ResNet-based backbone network to extract high-dimensional features from a target image of a set size and simultaneously obtain the two-dimensional key point coordinates of the target spacecraft in the image;

[0069] The multi-layer perceptron module uses the multi-layer perceptron (MLP) to predict the translation and rotation parameters of the target based on high-dimensional features and two-dimensional key point coordinates, and obtains the initial value of the target's pose distribution; the multi-layer perceptron outputs the probability distribution of the pose;

[0070] PnP pose optimization module: Using the correspondence between the coordinates of the two-dimensional key points and the three-dimensional coordinates of the target, the PnP method is used to perform nonlinear optimization on the initial value of the pose distribution, minimize the reprojection error, and obtain the six-degree-of-freedom parameters of the pose of the target spacecraft relative to the mother spacecraft, including three translation parameters and three rotation parameters;

[0071] The monocular camera is arranged on the mother spacecraft platform to photograph the target;

[0072] During the measurement system training process, the Monte Carlo method is used to sample a large number of predicted pose parameters and reproject the 3D points. That is, the target three-dimensional coordinates are reprojected according to the initial value of the pose distribution. The loss function is constructed based on the error between the two-dimensional coordinates obtained by the reprojection and the true two-dimensional coordinates to adjust the system network parameters.

[0073] Further, combined Figure 1 As shown in the figure, the preprocessing module can process a target 2D image of any resolution. It first converts the target 2D image into a three-channel grayscale image. Then, pixel normalization is performed, scaling the pixel intensities to a range of 0-1 to obtain a normalized grayscale image. During image acquisition, the monocular camera uses appropriate exposure settings to ensure image clarity and minimize noise. The image is then normalized to minimize the effects of varying lighting conditions. Through normalization and scaling, it adapts to the input requirements of the neural network.

[0074] The object detection module uses the object detection network (ODN) to efficiently identify the target region of interest (ROI) contained in the image, crop the ROI and adjust it to the standard size of 224x224 for network input to adapt to image input of any resolution, and ensure that only high-resolution processing is focused on key areas, reducing computational complexity while ensuring accuracy.

[0075] The key point regression module uses a residual network with ResNet as the backbone as a feature extractor to extract the high-dimensional features of the spacecraft image layer by layer, extract the features of the target image, and then input the extracted features into the key point regression network. Through the full convolution upsampling layer and self-attention layer, the key points on the spacecraft are finally located and the two-dimensional key point coordinates of the target are obtained. Unlike traditional methods, this key point detection based on deep learning can learn the high-order features of the target, avoiding the shortcomings of manually designed features, such as Figure 2 shown.

[0076] Combine Figure 2 As shown, the operations of the key point regression module on the target image of the set size include:

[0077] The target image of a set size is input into three compression excitation modules SE respectively. The outputs of the three compression excitation modules SE are simultaneously input into the four-fold full convolution upsampling module FCN-4s. The output of the four-fold full convolution upsampling module FCN-4s is sequentially input into the compression excitation module SE, the basic convolution module Conv-Basic, the two-dimensional maximum pooling module Maxpool2d, the basic convolution module Conv-Basic, the depth-separable convolution module Conv-DW and the compression excitation module SE, and then respectively input into three key point regression branches;

[0078] The modules that operate on the input data of each key point regression branch include the depth-separable convolution module Conv-DW, the basic convolution module Conv-Basic, the depth-separable convolution module Conv-DW, the compressed excitation module SE and the two-dimensional convolution module Conv2d.

[0079] The first key point regression branch obtains the two-dimensional key point coordinates of the target;

[0080] The high-dimensional feature obtained by the second key point regression branch is the target translation feature vector;

[0081] The high-dimensional feature obtained by the third key point regression branch is the target rotation feature vector;

[0082] When the key point regression module processes data, the modules with the same name that are passed through in sequence are different modules.

[0083] Combine Figure 3 As shown in the figure, the four-fold full convolution upsampling module FCN-4s includes three upsampling branches. The functions implemented by each upsampling branch include two-dimensional convolution Conv2d, two-dimensional batch normalization Batchnorm2d and ReLU activation.

[0084] The input of the two-dimensional convolution Conv2d of the three upsampling branches is the output of a corresponding compression excitation module;

[0085] The ReLU activated output of the first upsampling branch is used as the input of the two-dimensional transposed convolution ConvTranspose2d; the output of the two-dimensional transposed convolution ConvTranspose2d is added to the output of the ReLU activated output of the second upsampling branch, and then the output after the two-dimensional batch normalization Batchnorm2d, ReLU activation and two-dimensional transposed convolution ConvTranspose2d operation is added to the output of the ReLU activated output of the third upsampling branch. After two-dimensional batch normalization Batchnorm2d and ReLU activation, the output of the four-fold full convolution upsampling module is obtained as the input of the corresponding compressed excitation module in the key point regression module.

[0086] Combine Figure 4 As shown in the figure, the operations in the basic convolution module Conv-Basic include two-dimensional convolution Conv2d, two-dimensional batch normalization Batchnorm2d and ReLU activation in sequence.

[0087] Combine Figure 5 As shown in FIG, the operations in the compression excitation module SE include average pooling Avgpool, linear connection Linear, ReLU activation, linear connection Linear, ReLU activation and Sigmoid activation in sequence.

[0088] Combine Figure 6 As shown in the figure, the operations in the depth-wise separable convolution module Conv-DW include two-dimensional grouped convolution Conv2d (deepwise), two-dimensional batch normalization Batchnorm2d, ReLU activation, two-dimensional point-by-point convolution Conv2d (pointwise), two-dimensional batch normalization Batchnorm2d and ReLU activation.

[0089] The two-dimensional maximum pooling module Maxpool2d performs the following processing on the signal with image size (number of channels, width, height):

[0090] 1) The (3, 224, 224) image passes through the backbone network and outputs three sets of feature maps of different sizes: (128, 28, 28), (256, 14, 14), and (512, 7, 7).

[0091] 2) The feature maps of (128, 28, 28), (256, 14, 14), and (512, 7, 7) are each passed through a compression excitation module to strengthen the network's attention, and the output is still the feature maps of (128, 28, 28), (256, 14, 14), and (512, 7, 7).

[0092] 3) The feature maps of (128, 28, 28), (256, 14, 14), and (512, 7, 7) are upsampled by a 4x full convolution upsampling module, and the (14, 14) feature map and the (7, 7) feature map are upsampled and fused with the (28, 28) feature map to output a (256, 28, 28) feature map.

[0093] 4)(256, 28, 28) further strengthens the attention through a compression excitation module, and then passes through a basic convolution module with a stride of 2 and a maximum pooling module, the size is reduced to 1 / 4, the number of channels is doubled, and the feature map of (512, 7, 7) is output. It then passes through a basic convolution module, a depth-wise separable convolution module and a compression excitation module to further strengthen the features without changing the size and number of channels, and outputs a feature map of (512, 7, 7).

[0094] The (512, 7, 7) feature maps obtained by the two-dimensional maximum pooling module Maxpool2d are respectively fed into three independent regression prediction parts. Each part undergoes depth-separable convolution-basic convolution-depth-separable convolution-compression excitation. Finally, a two-dimensional convolution module without normalization is used to directly output feature maps to represent the predicted key point coordinates or feature vectors. The output feature map for predicting key point coordinates is (2*Num, 1, 1), rearranged into (Num, 2), representing the (x, y) coordinates of Num points. The output feature map for predicting translation or rotation feature vectors is (Num, 7, 7), rearranged into (Num, 49), representing the deep semantic information of translation or rotation contained in each point.

[0095] Going further, combined Figure 7 and Figure 8 As shown in Figure 1, the multilayer perceptron module predicts the translation and rotation parameters of the target from high-dimensional features and two-dimensional key point coordinates. The operation process includes:

[0096] The result of adding the two-dimensional key point coordinates of the target and the target translation feature vector is linearly connected Linear, one-dimensional batch normalization Batchnorm and LeakyReLU activation to obtain result one; at the same time, linear connection Linear, one-dimensional batch normalization Batchnorm and Sigmoid activation are performed to obtain result two; the result of multiplying result one and result two is linearly connected Linear, one-dimensional batch normalization Batchnorm, LeakyReLU activation, linear connection Linear, one-dimensional batch normalization Batchnorm, LeakyReLU activation and linear connection Linear in turn to obtain the mean of the Gaussian distribution of the target translation parameter;

[0097] The result of adding the two-dimensional key point coordinates of the target and the target rotation feature vector is linearly connected, one-dimensional batch normalization and LeakyReLU activation to obtain result three; at the same time, linear connection Linear, one-dimensional batch normalization Batchnorm and Sigmoid activation are performed to obtain result four; the result of multiplying result three and result four is linearly connected Linear, one-dimensional batch normalization Batchnorm, LeakyReLU activation, linear connection, one-dimensional batch normalization Batchnorm, LeakyReLU activation and linear connection Linear in turn to obtain the mean of the Gaussian distribution of the target rotation parameter;

[0098] The result of adding the two-dimensional key point coordinates of the target and the target translation feature vector is linearly connected, one-dimensional batch normalization and LeakyReLU activation to obtain result one; at the same time, linear connection Linear, one-dimensional batch normalization and Sigmoid activation are performed to obtain result two; the result of multiplying result one and result two is linearly connected, one-dimensional batch normalization, LeakyReLU activation, linear connection Linear, one-dimensional batch normalization, LeakyReLU activation and linear connection Linear in turn to obtain the Gaussian distribution covariance of the target translation parameter;

[0099] The result of adding the two-dimensional key point coordinates of the target and the target rotation feature vector is linearly connected, one-dimensional batch normalization Batchnorm and LeakyReLU activation to obtain result three; at the same time, linear connection Linear, one-dimensional batch normalization Batchnorm and Sigmoid activation are performed to obtain result four; the result of multiplying result three and result four is linearly connected Linear, one-dimensional batch normalization Batchnorm, LeakyReLU activation, linear connection Linear, one-dimensional batch normalization Batchnorm, LeakyReLU activation and linear connection Linear in turn to obtain the Gaussian distribution covariance of the target rotation parameter;

[0100] The initial value of the target's pose distribution is obtained by the mean of the target's translation parameter Gaussian distribution, the mean of the target's rotation parameter Gaussian distribution, the covariance of the target's translation parameter Gaussian distribution, and the covariance of the target's rotation parameter Gaussian distribution.

[0101] The multilayer perceptron module combines the features extracted from the keypoint regression network and the predicted two-dimensional coordinates and inputs them into the multilayer perceptron MLP to predict the probability distribution of the target's translation and rotation parameters. The MLP further captures the pose information of the spacecraft at different spatial positions, providing a reasonable initial estimate for PnP optimization, and at the same time represents the uncertainty of the translation and rotation parameters with probability distribution, such as Figure 7 and Figure 8 shown.

[0102] The details of information processing are as follows:

[0103] The key point coordinates of (Num, 1) (Num, 2) and the translation or rotation feature vector of (Num, 49) are combined and rearranged into a (51*Num)-dimensional input vector.

[0104] 2) The (51*Num)-dimensional input vector is linearly connected + one-dimensional batch normalization + LeakyReLU activation, and the output is a (2048)-dimensional intermediate vector. The (51*Num)-dimensional input vector is linearly connected + one-dimensional batch normalization + Sigmoid activation, and the output is a (2048)-dimensional intermediate vector.

[0105] 3) Multiply the above two (2048)-dimensional intermediate quantities point by point to transform them into a (2048)-dimensional intermediate vector to strengthen attention.

[0106] 4) The obtained (2048)-dimensional intermediate vector is repeatedly activated twice through linear connection + one-dimensional batch normalization + LeakyReLU. The number of channels is changed from (2048) dimensions to (1024) dimensions and then to (512) dimensions, gradually reducing the output intermediate vector.

[0107] 5) Finally, the (512)-dimensional intermediate vector is transformed into a Gaussian distribution of translation or rotation through a linear connection. The output mean vector includes the (3)-dimensional three-axis distance Gaussian distribution mean μt representing translation and the (4)-dimensional quaternion Gaussian distribution mean μr representing rotation. The output covariance vector includes the (3)-dimensional three-axis distance Gaussian distribution covariance ∑t representing translation and the (4)-dimensional quaternion Gaussian covariance vr representing rotation. μt and μr are the preliminary predicted poses t, r, Σt and Σr are used for subsequent sampling in the probability distribution.

[0108] Going further, PnP pose optimization: Using the known feature points on the target spacecraft 3D model and the corresponding 2D key points detected in the image, the Levenberg-Marquardt method is used to perform nonlinear optimization on the PnP preliminary solution to minimize the reprojection error and improve the accuracy of the pose estimation. The process is as follows:

[0109] 1) Given the coordinates of a 3D point (Num, 3)p3D, the coordinates of a 2D point (Num, 2)p2D, the initial three-axis distance (3)-dimensional vector t, and the initial quaternion (4)-dimensional vector r, define the objective function, i.e., the reprojection error, which represents the deviation between the 3D point after being mapped to the image plane by the current solution and the actual 2D point.

[0110] 2) For all 3D and 2D points, calculate the reprojected position of each point based on the current rotation and translation. Calculate the reprojection error for all points as the difference between the true position of the 2D point and the predicted position.

[0111] Combine Figure 1 As shown in Figure 2, the specific process of nonlinear optimization of the initial value of the pose distribution includes:

[0112] The two-dimensional coordinates obtained by reprojection are represented as projection2D:

[0113] projection2D=π(r·p3D+t),

[0114] Where π is the reprojection function, r is the initial rotation quaternion vector of the initial value of the pose distribution, p3D is the target three-dimensional coordinate, and t represents the translation parameter;

[0115] The error between the two-dimensional coordinates obtained by reprojection and the true two-dimensional coordinates is expressed as e:

[0116] e=∣∣p2D-projection2D∣∣ 2 ,

[0117] Where p2D represents the real two-dimensional coordinate;

[0118] Based on the error e, we differentiate r and t to generate the Jacobian matrix J.

[0119] Calculate the update step size Δθ based on the error e and the Jacobian matrix J, and use the Levenberg-Marquardt update formula to update the rotation quaternion (r+Δr) and translation vector (t+Δt), while keeping the quaternion normalized:

[0120] Δθ=(J T J+λI) -1 J T e,

[0121] Where λ is the damping coefficient;

[0122] Convergence judgment, check the change in error, or whether the maximum number of iterations has been reached. When the error e meets the set error threshold or the maximum number of iterations has been reached, stop the optimization and use the final optimization result as the six-degree-of-freedom parameter of the pose, including the updated quaternion r opt and the translation vector t opt ;

[0123] Reprojection error correction and optimization: Using the reprojection error as the loss function, the Monte Carlo method is used to sample the probability distribution of the predicted pose in large quantities, reproject the 3D points, compare the calculated error with the key points in the actual image, optimize the network parameters, and improve the accuracy and robustness of the pose estimation. The process is as follows:

[0124] 1) Randomly sample a set of translations t1, t2, ..., t from the translation Gaussian distribution N(μt, Σt) n , a set of quaternions r1, r2, ..., r are randomly sampled from the rotation Gaussian distribution N(μr, Σr) n , each sampling constitutes a complete pose (r1, t1), (r2, t2), ..., (r n , t n ), the sampling number n is 100 to 1000 which is a more reasonable value.

[0125] 2) For each set of sampled poses (r, t) i , calculate the reprojection error of all points.

[0126] projection2D i =π(r i p3D+t i ),

[0127] e i =|||p2D-projection2D i ∣∣ 2 ;

[0128] During the measurement system training process, the average E of all reprojection errors e is taken as the loss function:

[0129]

[0130] Where e i is the error corresponding to the i-th reprojection, and n is the number of reprojections;

[0131] E is used as the Monte Carlo loss to measure the automatic differentiation and gradient update of the system.

[0132] The experimental results show that the system of the present invention has the following advantages:

[0133] The system of the present invention has improved the key point extraction steps: compared with traditional key point extraction based on algorithms such as Harris, SIFT, and SURF, the system of the present invention has designed a key point regression network for feature extraction and key point regression, which can capture the complex texture and structural features of the spacecraft surface, especially when the lighting conditions change and the target surface has high-order details.

[0134] Regarding real-time performance and computational complexity: Due to its lightweight network structure and fast PnP solution, the proposed system boasts high computational speed, requiring only 9 GFLOPS per inference, meeting real-time requirements. Compared to binocular vision calculations and lidar point cloud registration, this implementation requires less data processing, making it suitable for the limited computing resources on board a spacecraft. To further enhance the model's real-time performance, network parameter quantization can be employed to reduce the model's computational overhead, thereby improving execution efficiency on embedded spacecraft platforms.

[0135] Advantages of Monocular Vision: This implementation significantly reduces hardware costs and system complexity by using monocular vision. Unlike stereo vision, monocular cameras do not require strict camera calibration and baseline length control, offering greater freedom in installation position and adapting to the complex environment of spacecraft.

[0136] Precision and robustness: Compared to ToF cameras, monocular cameras have higher planar resolution and can more accurately extract key features on spacecraft. By combining the optimization of reprojection errors, the accuracy of pose estimation is further improved. By simulating multiple lighting conditions during deep learning network training, this implementation can still stably extract feature points in scenes with significantly changing lighting conditions. In addition, combining the RANSAC algorithm to filter the detected key points can effectively suppress the influence of noise points and improve the robustness of pose estimation.

[0137] Application scenarios of the system of the present invention:

[0138] 1. Autonomous Spacecraft Rendezvous and Docking: This system is suitable for autonomous rendezvous and docking missions between spacecraft. Using a monocular camera, it acquires the position and pose of the target spacecraft in real time, guiding the mother spacecraft to perform precise relative displacement and attitude adjustments. During the initial rendezvous, the monocular camera can be used for long-range target detection and preliminary position estimation, providing foundational data for further rendezvous and approach.

[0139] 2. Space station target monitoring: The system of the present invention is suitable for monitoring target spacecraft in the space station. Through real-time posture measurement, it ensures the safety of the space station's docking target. The monocular vision system continuously obtains the attitude changes of the target spacecraft, providing early warning and target status monitoring for the space station.

[0140] 3. On-orbit servicing and maintenance: The system of the present invention is suitable for on-orbit servicing missions based on monocular vision key point detection and pose estimation, providing high-precision target position and pose data for maintenance tools such as robotic arms, thereby ensuring the accuracy and safety of maintenance operations.

[0141] The end-to-end pose output design based on probability distribution in the system of the present invention: the key point features are processed by a multi-layer perceptron to output the probability distribution of the pose of the target spacecraft, taking into account the uncertainty distribution of the pose and improving the description of the pose uncertainty.

[0142] Keypoint regression network design: Keypoint regression is performed on the features output by the residual network to improve the accuracy of spacecraft feature extraction, especially in complex lighting conditions and high-order detail features.

[0143] Loss function designed by combining probability distribution and Monte Carlo method: Based on the probability distribution of pose prediction, multiple samplings are performed in combination with the Monte Carlo method to construct a more appropriate loss function, thereby improving the robustness and accuracy of the model to the uncertainty of pose estimation.

[0144] The proposed system uses a direct, end-to-end approach for pose detection: high-order features are extracted through a convolutional neural network, combined with reprojection error optimization to improve pose estimation accuracy. The system directly outputs a probability distribution of pose as the measurement result, and then uses the Levenberg-Marquardt method for nonlinear optimization, avoiding the proxy loss caused by regressing only two-dimensional coordinate points for PnP calculation. It has been demonstrated that, using ESA's SPEEDPLUS monocular vision spacecraft image dataset, it achieves an average distance error of 0.085m and an average angle error of 3.4° with only minimal training.

[0145] Low computing resource consumption and strong real-time performance: Compared with lidar or binocular cameras, monocular cameras require less image processing data, making them suitable for the limited computing power of embedded platforms. A single inference requires only 9GFLOPS. After data quantization, the theoretical speed can exceed 100Hz on a 1TOPS NPU, which is much higher than lidar and can process spacecraft pose estimation in more real time.

[0146] The image input resolution is large and the observation distance is long: Compared with depth cameras or stereo vision, which are limited by hardware and have a short observation distance, monocular cameras can achieve an observation distance of 0.5 to 50 meters depending on the lens. The resolution depends on the sensing capability of the camera sensor and can reach millions to 20 million pixels.

[0147] Low hardware cost and complexity: Only a monocular camera is needed, without the need for complex sensor designs such as lidar or stereo vision, thus reducing the system hardware cost and structural complexity.

[0148] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the invention. It should be understood that many modifications may be made to the illustrative embodiments, and that other arrangements may be devised, without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in ways other than those described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in conjunction with other described embodiments.

Claims

1. A spacecraft posture measurement system based on monocular vision image processing, characterized in that: include: The preprocessing module is used to preprocess the target two-dimensional image captured by the monocular camera to obtain a normalized grayscale image; The target detection module obtains the region of interest containing the target from the normalized grayscale image and crops it into a target image of a set size; The key point regression module uses a ResNet-based backbone network to extract high-dimensional features from target images of a set size and obtain the two-dimensional key point coordinates of the target; The multi-layer perceptron module predicts the translation and rotation parameters of the target based on high-dimensional features and two-dimensional key point coordinates, and obtains the initial value of the target's pose distribution; PnP pose optimization module: Using the correspondence between the coordinates of the two-dimensional key points and the three-dimensional coordinates of the target, the PnP method is used to perform nonlinear optimization on the initial value of the pose distribution to obtain the six-degree-of-freedom parameters of the pose of the target spacecraft relative to the mother spacecraft; The monocular camera is arranged on the mother spacecraft platform; During the measurement system training process, the Monte Carlo method is used to reproject the target's three-dimensional coordinates according to the initial value of the pose distribution. The loss function is constructed based on the error between the two-dimensional coordinates obtained by the reprojection and the true two-dimensional coordinates to adjust the system network parameters. The operations of the key point regression module on the target image of the set size include: The target image of a set size is input into three compression excitation modules respectively, and the outputs of the three compression excitation modules are simultaneously input into the four-fold full convolution upsampling module. The output of the four-fold full convolution upsampling module is sequentially passed through the compression excitation module, the basic convolution module, the two-dimensional maximum pooling module, the basic convolution module, the depth-separable convolution module and the compression excitation module, and then respectively input into three key point regression branches; The modules that operate on the input data of each key point regression branch include depth-wise separable convolution module, basic convolution module, depth-wise separable convolution module, compressed excitation module and two-dimensional convolution module in sequence; The first key point regression branch obtains the two-dimensional key point coordinates of the target; The high-dimensional feature obtained by the second key point regression branch is the target translation feature vector; The high-dimensional feature obtained by the third key point regression branch is the target rotation feature vector; When the key point regression module processes data, the modules with the same name that are passed through in sequence are different modules.

2. The spacecraft posture measurement system based on monocular vision image processing according to claim 1, characterized in that: The preprocessing module first processes the target two-dimensional image into a three-channel grayscale image, and then performs pixel normalization to scale the pixel intensity to between 0 and 1 to obtain a normalized grayscale image.

3. The spacecraft posture measurement system based on monocular vision image processing according to claim 2, characterized in that: The key point regression module obtains the two-dimensional key point coordinates of the target through the fully convolutional upsampling layer and the self-attention layer.

4. The spacecraft posture measurement system based on monocular vision image processing according to claim 3, characterized in that: The quadruple full convolution upsampling module includes three upsampling branches. The functions implemented by each upsampling branch include two-dimensional convolution Conv2d, two-dimensional batch normalization Batchnorm2d and ReLU activation. The input of the two-dimensional convolution Conv2d of the three upsampling branches is the output of a corresponding compression excitation module; The ReLU activated output of the first upsampling branch is used as the input of the two-dimensional transposed convolution ConvTranspose2d; the output of the two-dimensional transposed convolution ConvTranspose2d is added to the output of the ReLU activated output of the second upsampling branch, and then the output after the two-dimensional batch normalization Batchnorm2d, ReLU activation and two-dimensional transposed convolution ConvTranspose2d operation is added to the output of the ReLU activated output of the third upsampling branch. After two-dimensional batch normalization Batchnorm2d and ReLU activation, the output of the four-fold full convolution upsampling module is obtained as the input of the corresponding compressed excitation module in the key point regression module.

5. The spacecraft posture measurement system based on monocular vision image processing according to claim 4, characterized in that: The operations in the basic convolution module include two-dimensional convolution Conv2d, two-dimensional batch normalization Batchnorm2d and ReLU activation in sequence.

6. The spacecraft posture measurement system based on monocular vision image processing according to claim 5, characterized in that: The operations in the compression excitation module include average pooling Avgpool, linear connection Linear, ReLU activation, linear connection Linear, ReLU activation and Sigmoid activation.

7. The spacecraft posture measurement system based on monocular vision image processing according to claim 6, characterized in that: The operations in the depthwise separable convolution module include 2D grouped convolution Conv2d, 2D batch normalization Batchnorm2d, ReLU activation, 2D point-wise convolution Conv2d, 2D batch normalization Batchnorm2d and ReLU activation in sequence.

8. The spacecraft posture measurement system based on monocular vision image processing according to claim 7, characterized in that: The operation process of the multi-layer perceptron module for predicting the translation parameters and rotation parameters of the target from the high-dimensional features and the two-dimensional key point coordinates includes: calculating the mean of the Gaussian distribution of the target translation parameters, the mean of the Gaussian distribution of the target rotation parameters, the covariance of the Gaussian distribution of the target translation parameters, and the covariance of the Gaussian distribution of the target rotation parameters; The initial value of the target's pose distribution is obtained by the mean of the target's translation parameter Gaussian distribution, the mean of the target's rotation parameter Gaussian distribution, the covariance of the target's translation parameter Gaussian distribution, and the covariance of the target's rotation parameter Gaussian distribution.

9. The spacecraft posture measurement system based on monocular vision image processing according to claim 8, characterized in that: The process of nonlinear optimization of the initial value of the pose distribution includes: The two-dimensional coordinates obtained by reprojection are represented as projection2D: projection2D=π(r·p3D+t), Where π is the reprojection function, r is the initial rotation quaternion vector of the initial value of the pose distribution, p3D is the target three-dimensional coordinate, and t represents the translation parameter; The error between the two-dimensional coordinates obtained by reprojection and the true two-dimensional coordinates is expressed as e: e=∣∣p2D-projection2D∣∣ 2 , Where p2D represents the real two-dimensional coordinate; Based on the error e, we differentiate r and t to generate the Jacobian matrix J. Calculate the update step size Δθ based on the error e and the Jacobian matrix J: Δθ=(J T J+λI) -1 J T e, Where λ is the damping coefficient; When the error e meets the set error threshold or the iteration reaches the maximum number of times, the final optimization result is used as the six-degree-of-freedom parameter of the pose; During the measurement system training process, the average E of all reprojection errors e is taken as the loss function: Where e i is the error corresponding to the i-th reprojection, and n is the number of reprojections; E is used as the Monte Carlo loss to measure the automatic differentiation and gradient update of the system.

Images