A Deep Learning-Based Method and System for Estimating the Relative Pose of Multi-Scale Spatial Targets in Complex Environments

Abstract

This invention relates to a method and system for estimating the relative pose of multi-scale spatial targets in complex environments based on deep learning, belonging to the field of spatial target pose estimation technology. This invention addresses the need for rapid pose calculation of spatial targets under highly dynamic and complex optical conditions in the space environment. Key technical points include: obtaining a pose sample set through virtual simulation of a 3D model of the spatial target at different positions and poses; constructing a 2D position regression network for target key points; inputting the pose sample set into the regression network to obtain a target training model; and solving for the pose using the random consistency perspective n-point method. This invention constructs a regression network for the 2D projection of key points of the spatial target based on deep learning. The network sample dataset images are input into a network composed of Darknet53 as the backbone. This invention constructs a regression network for the 2D position of key points of the spatial target, and then trains this network based on the pose sample set to obtain a training model capable of accurately predicting the target pose.

Description

Technical Field

[0001] This invention belongs to the field of spatial target pose estimation technology, specifically relating to a method and system for estimating the relative pose of multi-scale spatial targets in complex environments based on deep learning. Background Technology

[0002] With the rapid development of aerospace technology, the complexity and diversity of space missions are constantly increasing, and the demand for precise control of spacecraft is also growing stronger. These missions involve multiple aspects, such as autonomous rendezvous and docking of spacecraft, removal of space debris, and on-orbit maintenance of space equipment. All of these require accurate estimation of the pose (position and attitude) of space targets. The accuracy of pose estimation directly affects the efficiency and quality of mission execution, thus becoming one of the key technologies for the success of aerospace missions. In space target pose estimation technologies, traditional methods such as microwave radar and GPS technology are suitable for locating distant targets, while lidar and visual navigation technologies are more suitable for close-range observation. Although lidar technology has high accuracy, it is costly; while visual navigation technology, although less expensive, is susceptible to lighting conditions. Against this backdrop, monocular visual measurement technology, due to its advantages such as small sensor size, low power consumption, simple structure, and low cost, has become the preferred technology for measuring the pose of close-range targets in space.

[0003] In recent years, the rapid development of deep learning technology has brought new opportunities for spatial target pose estimation. Deep learning technologies, especially convolutional neural networks (CNNs), recurrent neural networks (RNNs), and graph neural networks (GNNs), have demonstrated outstanding performance in image recognition, target tracking, and scene understanding. These technologies can automatically learn useful features from complex, high-dimensional data, greatly improving the accuracy and efficiency of pose estimation. In particular, by training deep learning models on a large number of spatial scene images, the models can learn the complex features and pose change patterns of spatial targets, thereby achieving accurate prediction of the pose of unknown spatial targets. Furthermore, by using a two-stage method, combining spatial target model prediction features with prior information, the accuracy of pose estimation is further improved. With the introduction of deep learning technology, not only has the problem of drastic illumination changes encountered by traditional visual navigation technologies in spatial target pose estimation been solved, but its powerful learning and generalization capabilities have also provided new possibilities for improving the accuracy and efficiency of spatial target pose estimation. With the continuous progress and optimization of deep learning technology, it will show even greater potential and value in the field of spatial target pose estimation in the future.

[0004] The prior art (CN113034581A) discloses a deep learning-based method for estimating the relative pose of spatial targets, which includes the following steps: a) constructing a labeled sample set using the two-dimensional projection of the three-dimensional model of the spatial target at different positions and orientations; b) dividing the labeled sample set into training, validation, and test sets, and constructing a pose estimation neural network; c) inputting the training and validation sets into the constructed pose estimation neural network for training to obtain a pose estimation model; d) using the trained pose estimation model to test the test set to obtain the pose information of the spatial target for each sample in the test set. This prior art enables the simultaneous estimation of the position and orientation information of a spatial target using a regression model of a single image, and is suitable for target pose estimation under complex lighting conditions in space. However, the accuracy of pose estimation for targets in the space environment still needs further improvement. Summary of the Invention

[0005] The technical problem to be solved by this invention is:

[0006] To address the challenges of complex optical conditions and rapid pose estimation for space targets in highly dynamic space environments, and to improve the accuracy of target pose estimation in space environments, this invention proposes a method and system for estimating the relative pose of multi-scale space targets in complex environments based on deep learning (deep learning network algorithms).

[0007] The technical concept of this invention is as follows: After establishing a sample set using rendering engine software (such as Unity, UrealEngine, OpenGL software, or other existing software with this function, the above software is selected for trial use for scientific research purposes), image features are extracted using a multi-layer neural network, the image plane projection point position of the predefined three-dimensional point is regressed, and the predicted two-dimensional position of the three-dimensional point is obtained. Then, the target motion is solved using the random consistency perspective n-point method to achieve six-degree-of-freedom pose estimation of the spatial target.

[0008] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0009] A method for estimating the relative pose of multi-scale spatial targets in complex environments based on deep learning is described below.

[0010] Step 1: Based on the virtual simulation of the 3D model of the spatial target under different positions and attitudes, obtain the pose sample set:

[0011] Import open-source 3D models of spatial targets into rendering engine software (such as Unity, UrealEngine, and OpenGL software), simulate different spatial positions and poses of the spatial targets, and use a K-camera with set intrinsic parameters for shooting.

[0012] The pose sample set contains spatial target pose change data, which exists in the form of pose transformation matrix R and position translation vector T; the sample set contains image data of spatial targets observed from multiple angles and distances, covering a wide range of lighting conditions and environmental backgrounds.

[0013] Step 2: Construct a two-dimensional location regression network for the target key points:

[0014] A regression network for the two-dimensional projection of key points of spatial targets is constructed. The network sample dataset images (from the dataset images in step one) are input into the network constructed with Darknet53 as the backbone network. Then, the downsampled image features are passed to the FPN feature pyramid network, followed by upsampling processing, regression of the two-dimensional projection of key points, and finally feature fusion to obtain the predicted two-dimensional position of key points.

[0015] Step 3: Input the pose sample set into the regression network to obtain the target training model:

[0016] After constructing a regression network for the two-dimensional position of key points of a spatial target, the network is trained based on a pose sample set to obtain a training model that can accurately predict the pose of the target. The key to the training setup in this process is to ensure that the model can learn all possible poses of the spatial target from the pose sample set, thereby achieving accurate prediction of unknown poses.

[0017] Step 4: Solve the pose using the randomized consistent perspective n-point method.

[0018] After obtaining the two-dimensional projection of the key points, and combining the coordinates of the key points in the target body coordinate system, control points are first selected, and other points are represented as linear combinations of these control points.

[0019] The following steps are performed: The relative pose between the target body coordinate system and the camera coordinate system is obtained using the random consistent perspective n-point method. Then, a linear equation about the position of the control point in the camera coordinate system is constructed. The optimal solution is found using singular value decomposition (SVD), and the position of the control point in the camera coordinate system is obtained. By comparing the positions of the control point in the world coordinate system and the camera coordinate system, the rotation and translation matrices of the camera are calculated (the relative pose of the spatial target refers to the relative position and attitude, which are expressed in the attitude rotation matrix and translation matrix).

[0020] Furthermore, in step one, various disturbances that may be encountered in the space environment are simulated, including changes in illumination, shadows, reflections, and atmospheric disturbances, to enhance the model's generalization ability and robustness.

[0021] Furthermore, camera parameters and spatial target 3D model parameters are set in the rendering engine software. A 2D image (pose sample set) is generated by imaging the spatial target 3D model using the camera. The camera parameters include the camera distance S from the spatial target, the camera's intrinsic field of view θ, and the width and height of the spatial target image. The spatial target 3D model parameters include the spatial target 3D model's position (x, y, z), the spatial target 3D model's three-axis attitude angles (α, θ, ψ), and its rotational angular velocity.

[0022] Furthermore, the specific implementation process of step two is as follows:

[0023] (1) The image is entered into the backbone network for downsampling processing.

[0024] First, the spatial target images from the aforementioned pose sample set are input into the backbone neural network for downsampling. The backbone network output features are divided into five stages, each representing a feature vector at a different scale. The backbone network structure consists of five main stages, each composed of a series of convolutional layers and max-pooling layers, used to extract and process image features at different levels. In the first stage, the network begins preliminary feature extraction and downsampling of the input image using a 7x7 convolutional kernel (2x2 stride, 64 channels) and a 3x3 max-pooling layer (2x2 stride). Next, the second stage consists of a convolutional block (conv_blo The first stage consists of one convolutional block and two identity blocks, with all blocks configured with 64, 64, and 256 channels to further deepen feature understanding. The second stage includes one convolutional block and four identity blocks, with their channel counts increased to 128, 128, and 512 to capture more complex features. In the third stage, the network explores features more deeply through one convolutional block and eight identity blocks (with channel counts of 256, 256, and 1024). Finally, the fifth stage consists of one convolutional block and two identity blocks, with the largest number of channels, reaching 512, 512, and 2048, to complete the in-depth analysis and understanding of image features.

[0025] (2) Upsampling of feature vectors in multi-scale feature networks

[0026] After downsampling by the backbone network, the resulting feature vectors are then fed into the FPN (Feature Pyramid Network) for upsampling. The FPN uses a bottom-up and lateral connection structure to construct feature maps of different scales in order to capture information from coarse to fine in the image. In this process, the lower-level high-resolution feature maps are upsampled and fused with the feature maps of the upper level, gradually achieving feature refinement and scale unification.

[0027] (3) Multi-scale feature fusion processing of randomized vectors

[0028] After completing the upsampling of the feature vectors, a randomized fusion strategy is then used to perform multi-scale feature fusion of the randomized vectors to enhance the generalization ability and robustness of the model. By randomizing the feature maps at different scales and then combining the processed feature maps, the diversity and variability of spatial targets are captured to ensure the adaptability of different spatial targets and provide richer and more accurate feature information for subsequent key point location prediction.

[0029] (4) Regression generates two-dimensional location prediction of key points

[0030] To predict the two-dimensional location of key points of spatial targets, regression is performed using feature maps that have undergone multi-scale feature fusion. A regression network is constructed and trained to learn the mapping relationship from the feature map to the two-dimensional location of the key points. In the final stage of the network, fully connected layers or convolutional layers are used to directly perform regression prediction on the two-dimensional location of each key point. By carefully adjusting the network parameters and structure, high-precision prediction of the location of key points of spatial targets is achieved.

[0031] Furthermore, the detailed training settings for step three are as follows:

[0032] (1) Loss function selection: Use a loss function suitable for the pose regression task, such as mean squared error (MSE) or a customized loss function that is more suitable for the specific task. To accurately measure the difference between the predicted pose and the true pose;

[0033] (2) Optimizer selection: Select the efficient optimizer Adam and adjust the appropriate learning rate to ensure the stability and convergence speed of the training process; at the same time, set the batch size and training epochs reasonably to balance training time and model performance.

[0034] (2) Early Stopping: Monitor the performance on the validation set. When the model no longer improves after a certain number of rounds, stop training to avoid overfitting. After stopping training, jump to step four to solve the pose.

[0035] Furthermore, the specific process of multi-scale feature fusion processing of randomized vectors is as follows:

[0036] Pixel information of the region where the spatial target is located participates in the prediction with certain weights, s k The feature map sizes at different layers of the network are selected based on the object size distribution in the target dataset. S is the size of the longer side of the target's two-dimensional bounding box in the original image, and N is the weight of the k-th layer. k satisfy

[0037]

[0038] in

[0039]

[0040] By utilizing information from all levels within the feature pyramid network, the robustness of the network to changes in target shooting distance is enhanced, thereby improving the overall performance and accuracy of target pose estimation.

[0041] Furthermore, step three specifically involves:

[0042] After constructing the 2D point regression network, the pose sample set obtained in step one is input into the network for training. The model with the smallest loss function is selected as the final training result. Before training, the network's loss function needs to be designed. For a 3D point p... i According to its projection process:

[0043]

[0044] make

[0045]

[0046]

[0047] The true two-dimensional position of the projection point The epipolar direction passing through the real point, where K is the camera intrinsic parameter matrix, is the 3D keypoint p corresponding to the camera ray. i In the camera coordinate system, where R is the ground-truth rotation matrix and T is the translation vector; the reprojection error is calculated and mapped to 3D space to obtain:

[0048]

[0049] in

[0050]

[0051] It is related to camera rays The matrix of 3D points from orthogonal projection; finally, the loss is:

[0052]

[0053] Furthermore, the specific process of step four is as follows:

[0054] After obtaining the two-dimensional projected coordinates of eight keypoints using a two-dimensional point regression network, the pose is solved using the random consistency perspective n-point method. The random consistency perspective n-point method consists of a random consistency method and a perspective n-point method. The random consistency method includes three assumptions: the data consists of "interior points" and these "interior point" data can be interpreted using mathematical model parameters; the data consists of "outterior points"; and the method process is as follows:

[0055] (1) Determine the minimum sample set n required for mathematical model calculation and randomly select M minimum sample sets from all sample data P, and use these M minimum sample sets to calculate the parameters of the mathematical model respectively;

[0056] (2) Calculate the error with these models using the remaining data, and identify the data as an “interior point” of the current mathematical model when the error is less than a given threshold;

[0057] (3) Count the number of “interior points” under M mathematical models. The model with the most “interior points” is the correct mathematical model. Then, use the least squares method to obtain the accurate mathematical model using all the “interior point” data under the model.

[0058] (4) If the corresponding set of "interior points" is not found after a certain number of samplings, the algorithm is considered to have failed; otherwise, the result obtained in step 3 shall be taken as the final result.

[0059] Four of the eight corner points regressed in step (2) are selected as control points. The coordinates of these points in the target body coordinate system are known, and the two-dimensional positions of these points in the image have been obtained.

[0060] All 3D points are then represented as linear combinations of these four control points; each 3D point can be represented by four control points and a set of weights, which reflect the relative position between the point and the control points, as shown in the following formula, where the 3D coordinates P of a series of 8 key points in the body coordinate system are already known. i w i = 1…8

[0061]

[0062] in, Let α be the three-dimensional coordinates of the control point in the world system. ij The control coefficients are constructed by using the camera model and the representations of the 3D points obtained in the previous steps to create a linear equation system P for the positions of the control points in the camera coordinate system. i w The unknowns in the system are the coordinates of the control points and the intrinsic parameters of the camera.

[0063] By solving the linear system, the positions of the control points in the camera coordinate system are obtained; this step usually involves matrix operations, and singular value decomposition (SVD) is used to find the optimal solution.

[0064] The positions of the control points in the camera coordinate system were obtained. By comparing their positions in the world coordinate system and the camera coordinate system, the rotation and translation matrices of the camera were calculated.

[0065] A deep learning-based system for estimating the relative pose of multi-scale spatial targets in complex environments is provided. This system has program modules corresponding to the steps of the aforementioned deep learning-based method for estimating the relative pose of multi-scale spatial targets in complex environments. When running, the system executes the steps in the deep learning-based method for estimating the relative pose of multi-scale spatial targets in complex environments.

[0066] A computer-readable storage medium storing a computer program configured to, when invoked by a processor, implement the steps of a deep learning-based method for estimating the relative pose of a multi-scale spatial target in a complex environment.

[0067] The present invention has the following beneficial technical effects:

[0068] This invention provides an efficient and accurate solution for spatial target pose estimation using a deep learning network algorithm. By employing an encoder-decoder structure, this algorithm can deeply mine the rich information in image data, achieving high-precision estimation of the six-DOF pose of spatial targets. The effects and advantages of this invention are described in detail below:

[0069] (1) Stability under high dynamic complex optical conditions

[0070] Optical conditions in the space environment are complex and variable, including direct sunlight, reflected light from the Earth, and interference from other artificial light sources. These conditions pose a significant challenge to target pose estimation in space. The algorithm in this invention effectively extracts key features of the target under different lighting conditions by fusing the regression results from each layer of the FPN network. This enhances the model's adaptability to complex optical environments and ensures accurate target pose estimation even in highly dynamic space environments.

[0071] (2) Overcoming the recognition problem at different distances and scales

[0072] Traditional spatial target pose estimation methods show significant differences in performance when dealing with targets at different distances and scales, especially for distant targets, where identification and localization are more challenging. This invention integrates the feature layers of an FPN network, enabling the model to utilize image information at different scales simultaneously. This improves the accuracy of target identification across various distances and scales, ensuring both the accuracy and robustness of pose estimation.

[0073] (3) Accurate three-dimensional point pose calculation

[0074] After extracting image features using a multi-layer neural network, the algorithm performs regression analysis on the image plane projection points of predefined 3D points. Then, it employs a randomized consistent perspective n-point method to solve for the target motion, achieving high-precision six-DOF pose estimation of spatial targets. Compared to traditional calculation techniques, this method can more accurately reflect the actual position and attitude of spatial targets, improving the accuracy and efficiency of pose estimation.

[0075] (4) Innovation in error assessment methods

[0076] This invention employs an error assessment method that reprojects two-dimensional points into three-dimensional space. This method effectively overcomes the problem of traditional error assessment being affected by target distance. Through this method, the algorithm can more accurately assess the pose estimation error, thereby further optimizing model performance and ensuring the high reliability of the pose estimation results.

[0077] (5) Test performance

[0078] In the final test set, the algorithm of this invention demonstrated excellent performance, with an average three-axis Euler angle error of only 1.92° and a maximum value not exceeding 5°, showing the high accuracy and stability of the model in space target pose estimation. These test results fully demonstrate the efficiency and reliability of the algorithm of this invention in practical applications, providing solid technical support for the execution of future space missions. Attached Figure Description

[0079] Figure 1 Here is a flowchart of the pose estimation network. Figure 2 This is a diagram illustrating the satellite's on-orbit status. Figure 3 The dataset images are labeled with spatial target bounding boxes. Figure 4 is a flowchart of the pose estimation network structure, Figure 5 is a diagram of the Darknet53 structure, and Figure 6 is a diagram of the multi-scale feature pyramid network structure. Figure 7 For the target pose estimation map, Figure 8 Figure 9 shows the test error diagrams for each axis (x, y, z). Figure 9 is a set of error tracking diagrams, in which: (a) is the error tracking diagram for the Z-axis, (b) is the error tracking diagram for the Y-axis, and (c) is the error tracking diagram for the Z-axis. Detailed Implementation

[0080] Combined with appendix Figure 1 -9 The implementation of the present invention is described as follows:

[0081] This application provides a deep learning method and system for multi-scale spatial target relative pose estimation in complex environments, based on deep learning. The method includes the following steps:

[0082] Step 1: Obtain a pose sample set by virtually simulating the 3D model of the spatial target under different positions and attitudes.

[0083] This step uses the virtual technology provided by Unity simulation software to construct a 3D model of the space target and simulates the space target at different spatial positions and orientations. By rendering these 3D models with high realism and using a camera with set intrinsic parameters K to capture images, a pose sample set containing rich space target pose change data exists in the form of an orientation transformation matrix R and a position translation vector T. This sample set includes image data of the space target observed from multiple angles and distances, covering a wide range of lighting conditions and environmental backgrounds. Furthermore, this step also simulates various disturbances that may be encountered in the space environment, such as lighting changes, shadows, reflections, and atmospheric disturbances, to enhance the model's generalization ability and robustness.

[0084] The shooting software is also based on Unity for virtual simulation rendering. In Unity, camera parameters and spatial target 3D model parameters are set, and the camera images the spatial target 3D model to generate a 2D image. The camera parameters include the distance S of the camera from the spatial target, the camera intrinsic field of view θ, and the width and height of the spatial target image. The spatial target 3D model parameters include the position (x, y, z) of the spatial target 3D model, the three-axis attitude angles (α, θ, ψ) of the spatial target 3D model, and its rotational angular velocity (α&, θ&, ψ&).

[0085] Step 2: Construct a two-dimensional location regression network for target key points

[0086] This invention constructs a regression network for the two-dimensional projection of key points of spatial targets based on deep learning. The network sample dataset images are input into a network composed of Darknet53 as the backbone network. Then, the downsampled image features are passed to the FPN feature pyramid network, followed by upsampling processing. Finally, the two-dimensional projection of key points is regressed through feature fusion to obtain the predicted two-dimensional position of key points.

[0087] (1) The image is entered into the backbone network for downsampling processing.

[0088] First, the spatial target images from the aforementioned sample set are input into the backbone neural network for downsampling. The backbone network output features are divided into five stages, representing feature vectors of five different scales. The backbone network structure consists of five main stages, each composed of a series of convolutional layers and max-pooling layers, specifically designed to extract and process image features at different levels. In the first stage, the network begins preliminary feature extraction and downsampling of the input image using a 7x7 convolutional kernel (2x2 stride, 64 channels) and a 3x3 max-pooling layer (2x2 stride). Next, the second stage consists of a convolutional block (conv_block) and two identity blocks (identity_blocks), with the number of channels configured as 64, 64, and 256 respectively, further deepening feature understanding. The third stage includes a convolutional block and four identity blocks, with their channel numbers increased to 128, 128, and 512 respectively, to capture more complex features. In the fourth stage, the network explores features more deeply through one convolutional block and eight identity blocks (with 256, 256, and 1024 channels). Finally, the fifth stage consists of one convolutional block and two identity blocks, with the largest number of channels, reaching 512, 512, and 2048, to complete the in-depth analysis and understanding of image features.

[0089] (2) Upsampling of feature vectors in multi-scale feature networks

[0090] According to one aspect of the present invention, after downsampling processing by the backbone network, the resulting feature vector is then fed into an FPN (Feature Pyramid Network) for upsampling processing. The FPN utilizes a bottom-up and laterally connected structure to construct feature maps of different scales, thereby capturing information from coarse to fine in the image. During this process, lower-level high-resolution feature maps are upsampled and fused with higher-level feature maps, gradually achieving feature refinement and scale unification.

[0091] (3) Multi-scale feature fusion processing of randomized vectors

[0092] In this invention, after upsampling the feature vectors, the next step is multi-scale feature fusion processing of randomized vectors. This step employs a randomized fusion strategy to enhance the model's generalization ability and robustness. By randomizing feature maps at different scales and then combining these processed feature maps, the diversity and variability of spatial targets can be captured more effectively. This processing ensures adaptability to different spatial targets and provides richer and more accurate feature information for subsequent keypoint location prediction.

[0093] (4) Regression generates two-dimensional location prediction of key points

[0094] In this invention, the prediction of the two-dimensional location of key points of spatial targets is achieved by regression using feature maps that have undergone multi-scale feature fusion processing. The regression network constructed here learns the mapping relationship from the feature map to the two-dimensional location of the key points through training. In the final stage of the network, fully connected layers or convolutional layers are used to directly perform regression prediction on the two-dimensional location of each key point. By carefully adjusting the network parameters and structure, high-precision prediction of the location of key points of spatial targets can be achieved.

[0095] Step 3: Input the pose sample set into the regression network to obtain the target training model.

[0096] This invention constructs a regression network for the two-dimensional positions of key points of spatial targets. This network is then trained using a pose sample set to obtain a trained model capable of accurately predicting target poses. The key to this training setup is ensuring that the model can learn all possible poses of the spatial target from the pose sample set, thereby achieving accurate prediction of unknown poses. The following is a detailed training setup for this step:

[0097] (1) Loss function selection: Use a loss function suitable for the pose regression task, such as mean squared error (MSE) or a customized loss function that is more suitable for the specific task, in order to accurately measure the difference between the predicted pose and the true pose.

[0098] (2) Optimizer selection: Select the efficient optimizer Adam and adjust the learning rate appropriately to ensure the stability and convergence speed of the training process. At the same time, set the batch size and training epochs reasonably to balance training time and model performance.

[0099] (2) Early Stopping: Monitor the performance on the validation set and stop training when the model’s performance no longer improves after a certain number of rounds to avoid overfitting.

[0100] Step 4: Solve the pose using the randomized consistent perspective n-point method.

[0101] After obtaining the 2D projection of the key points, and combining the coordinates of the key points in the target body coordinate system, control points are first selected, and other points are represented as linear combinations of these control points. Then, the relative pose between the target body coordinate system and the camera coordinate system is obtained using the random consistent perspective n-point method. A linear equation about the position of the control points in the camera coordinate system is then constructed, and the optimal solution is found using singular value decomposition (SVD). The position of the control points in the camera coordinate system is obtained. By comparing their positions in the world coordinate system and the camera coordinate system, the rotation and translation matrix of the camera is calculated.

[0102] Example:

[0103] The following is based on the appendix Figure 1-9 provides a detailed description of the present invention.

[0104] A flowchart illustrating the spatial target relative pose estimation method and system construction method based on deep learning is shown below. Figure 1 As shown.

[0105] Step 1: Obtain a pose sample set by virtually simulating the 3D model of the spatial target under different positions and attitudes.

[0106] Establishing a three-dimensional model of a space target in orbit Figure 2 As shown, the spatial target relative pose estimation method based on deep learning of the present invention first performs key point bounding box annotation on the spatial target of the 3D model, specifically selecting the satellite body for bounding box annotation. Figure 3 As shown, the pose data sample dataset is output through the pose transformation relationship between the bounding box position and the camera coordinate system, and the position translation vector. Then, the labeled sample set is divided into training, validation, and test sets, and a pose estimation neural network is constructed. The ratio of the training, validation, and test sets in the dataset is shown in Table 1 below.

[0107] Table 1 Image dataset for spatial target relative pose estimation

[0108] training set test set Validation set 35,000 2500 sheets 2500 sheets

[0109] Step 2: Construct a two-dimensional location regression network for target key points

[0110] This invention constructs a regression network for the two-dimensional projection of key points of spatial targets based on deep learning. The network sample dataset images are input into a network composed of Darknet53 as the backbone network. Then, the downsampled image features are passed to the FPN feature pyramid network, followed by upsampling processing to regress the two-dimensional projection of key points. Finally, the predicted two-dimensional positions of key points are obtained through feature fusion.

[0111] (1) The image is entered into the backbone network for downsampling processing.

[0112] This invention designs a deep learning model based on the Darknet53 backbone network to extract key features from satellite images and predict the two-dimensional positions of satellite key points. A schematic diagram of the specific backbone network structure is shown in Figure 5.

[0113] First, the spatial target images from the aforementioned sample set are input into the backbone neural network for downsampling processing. The backbone network output features are divided into five stages, each representing a feature vector at a different scale. The backbone network structure consists of five main stages, each composed of a series of convolutional layers and max-pooling layers, specifically designed to extract and process image features at different levels.

[0114] ① Stage 1 Operation: The image first enters the backbone network and undergoes downsampling. Convolutional Layer: A 7x7 convolutional kernel is used for convolution, with a stride of 2x2 and 64 channels. This design helps to initially extract the basic features of the image. Max Pooling Layer: Immediately following the convolutional layer, a 3x3 max pooling layer is used for further downsampling, also with a stride of 2x2. This helps reduce feature dimensionality while retaining important feature information.

[0115] ② Stage 2 Structure: This stage consists of one convolutional block and two identity blocks. Channel configuration: In this stage, the number of channels in the convolutional block and the identity blocks are configured as 64, 64, and 256, respectively. This design deepens the network's understanding of image features and improves its feature representation ability.

[0116] ③ Stage 3 Structure: This stage consists of one convolutional block and four identity blocks. Channel Count Increase: The number of channels in this stage is increased to 128, 128, and 512 to capture more complex and detailed features. With more identity blocks, the network is able to learn deeper feature representations.

[0117] ④ Stage 4: Consists of one convolutional block and eight identity blocks. Number of channels: The number of channels is further increased to 256, 256, and 1024. This stage aims to explore image features through a deeper network structure to achieve accurate capture of target details.

[0118] Stage 5 consists of a convolutional block and two identity blocks. Maximum number of channels: In this stage, the number of channels reaches its maximum, set to 512, 512, 2048. This design achieves in-depth analysis and understanding of image features, providing ample feature support for accurate keypoint localization.

[0119] (2) Upsampling of feature vectors in multi-scale feature networks

[0120] According to one aspect of the present invention, after downsampling processing by the backbone network, the resulting feature vector is then fed into an FPN (Feature Pyramid Network) for upsampling processing. The structure of the FPN is shown in Figure 6. First, there is a bottom-up path structure. This path gradually reduces the spatial size of the feature map while increasing its depth through a series of convolutional and pooling layers, thereby capturing features from shallow to deep layers. Then, there is a top-down path. This top-down path starts from the deepest feature map output by the bottom-up path and gradually enlarges the feature map to fit the size of shallower feature maps.

[0121] (3) Multi-scale feature fusion processing of randomized vectors

[0122] This invention employs weighted fusion processing on feature maps of different scales, and then combines these processed feature maps to more effectively capture the diversity and variability of spatial targets. This method allows pixel information from the target's region to participate in prediction with certain weights. k The feature map sizes for different layers of the network are selected based on the object size distribution in the target dataset. S is the size of the longer side of the target's two-dimensional bounding box in the original image. The weights N of the k-th layer are... k satisfy

[0123]

[0124] in

[0125]

[0126] By utilizing information from all levels within the feature pyramid network, the robustness of the network to changes in target shooting distance is enhanced, thereby improving the overall performance and accuracy of target pose estimation.

[0127] (4) Regression generates two-dimensional location prediction of key points

[0128] In this invention, the prediction of the two-dimensional location of key points of spatial targets is achieved by regression using feature maps that have undergone multi-scale feature fusion processing. The regression network constructed here learns the mapping relationship from the feature map to the two-dimensional location of the key points through training. In the final stage of the network, fully connected layers or convolutional layers are used to directly perform regression prediction on the two-dimensional location of each key point. By carefully adjusting the network parameters and structure, high-precision prediction of the location of key points of spatial targets can be achieved.

[0129] Step 3: Input the pose sample set into the regression network to obtain the target training model.

[0130] After constructing the 2D point regression network, the pose sample set obtained in step one is input into the network for training. The model with the smallest loss function is selected as the final training model. Before training, the network's loss function needs to be designed. For a 3D point p... i According to its projection process:

[0131]

[0132] make

[0133]

[0134] The true two-dimensional position of the projection point The epipolar direction passing through the real point, where K is the camera intrinsic parameter matrix, is the 3D keypoint p corresponding to the camera ray. iIn the camera coordinate system, where R is the ground-truth rotation matrix and T is the translation vector; the reprojection error is calculated and mapped to 3D space to obtain:

[0135]

[0136] in

[0137]

[0138] It is a camera ray The matrix of 3D points from orthogonal projection, and finally, the loss is taken as

[0139]

[0140] The hyperparameters of the pose estimation network in this invention are set as shown in Table 2 below:

[0141] Table 3. Hyperparameters of the Corner Position Regression Network

[0142]

[0143] Step 4: Solve the pose using the randomized consistent perspective n-point method.

[0144] After obtaining the two-dimensional projected coordinates of eight keypoints using a two-dimensional point regression network, the pose is solved using the random consistency perspective n-point method. The random consistency perspective n-point method consists of a random consistency method and a perspective n-point method. The random consistency method includes three assumptions: the data consists of "interior points" and these "interior point" data can be explained by mathematical model parameters; the data consists of "outterior points" and these "outterior point" data cannot be used with a mathematical model; and the data contains noisy data. The method process is as follows:

[0145] (1) Determine the minimum sample set n required for mathematical model calculation and randomly select M minimum sample sets from all sample data P. Use these M minimum sample sets to calculate the parameters of the mathematical model respectively.

[0146] (2) Calculate the error with these models using the remaining data, and identify the data as an “interior point” of the current mathematical model when the error is less than a given threshold.

[0147] (3) Count the number of “interior points” under M mathematical models. The model with the most “interior points” is the correct mathematical model. Then, use the least squares method to obtain the accurate mathematical model using all the “interior point” data under the model.

[0148] (4) If the corresponding set of “interior points” is not found after a certain number of samplings, the algorithm is considered to have failed; otherwise, the result obtained in step 3 is taken as the final result.

[0149] Four of the eight corner points regressed in step two are selected as control points. The coordinates of these points in the target body coordinate system are known, and the two-dimensional positions of these points in the image have been obtained.

[0150] All 3D points can then be represented as linear combinations of these four control points. This means that each 3D point can be represented by four control points and a set of weights, which reflect the relative position between the point and the control points, as shown in the following equation, where the 3D coordinates P of a series of 8 key points in the body coordinate system are already known. i w i = 1…8

[0151]

[0152] in, Let α be the three-dimensional coordinates of the control point in the world system. ij The control coefficients are constructed by using the camera model and the representations of the 3D points obtained in the previous steps to create a linear equation system P for the positions of the control points in the camera coordinate system. i w The unknowns in the system are the coordinates of the control points and the intrinsic parameters of the camera.

[0153] By solving the linear system, the positions of the control points in the camera coordinate system are obtained. This step typically involves matrix operations, using singular value decomposition (SVD) to find the optimal solution.

[0154] The positions of the control points in the camera coordinate system were obtained. By comparing their positions in the world coordinate system and the camera coordinate system, the rotation and translation matrices of the camera were calculated.

Claims

1. A method for estimating the relative pose of multi-scale spatial targets in complex environments based on deep learning, characterized in that, The implementation process of the method is as follows: Step 1: Obtain a pose sample set by virtually simulating the 3D model of the spatial target under different positions and attitudes. Import an open-source 3D model of a spatial target into the rendering engine software, simulate the different spatial positions and postures of the spatial target, and take pictures using a K-camera with set intrinsic parameters. The pose sample set contains spatial target pose change data, which exists in the form of pose transformation matrix R and position translation vector T; the sample set contains image data of spatial targets observed from multiple angles and distances, covering a wide range of lighting conditions and environmental backgrounds. Step 2: Construct a regression network for the two-dimensional locations of key points of spatial targets: A regression network for the two-dimensional location of key points of spatial targets is constructed. The network sample dataset images are input into the network with Darknet53 as the backbone network. Then, the downsampled image features are passed to the FPN feature pyramid network, followed by upsampling processing. The two-dimensional projection of the key points is regressed, and finally the predicted two-dimensional location of the key points is obtained through feature fusion. Step 3: Input the pose sample set into the regression network to obtain the target training model: After constructing a regression network for the two-dimensional position of key points of a spatial target, the network is trained based on a pose sample set to obtain a training model that can accurately predict the pose of the target. The key to the training setup in this process is to ensure that the model can learn all possible poses of the spatial target from the pose sample set, thereby achieving accurate prediction of unknown poses. Detailed training settings: (1) Loss function selection: Use the mean squared error (MSE) loss function suitable for the pose regression task or a customized loss function more suitable for the specific task to accurately measure the difference between the predicted pose and the true pose. (2) Optimizer selection: Select the efficient optimizer Adam and adjust the appropriate learning rate to ensure the stability and convergence speed of the training process; at the same time, set the batch size and training epochs reasonably to balance training time and model performance. (2) Early Stopping: Monitor the performance on the validation set. When the model no longer improves after a certain number of rounds, stop training to avoid overfitting. After stopping training, jump to step four to solve the pose. Step 4: Solve the pose using the randomized consistent perspective n-point method. After obtaining the two-dimensional projection of the key points, and combining the coordinates of the key points in the target body coordinate system, control points are first selected, and other points are represented as linear combinations of these control points. The following steps are performed: The relative pose between the target body coordinate system and the camera coordinate system is obtained using the random consistent perspective n-point method. Then, a linear equation about the position of the control point in the camera coordinate system is constructed. The optimal solution is found using singular value decomposition (SVD), and the position of the control point in the camera coordinate system is obtained. By comparing the positions of the control point in the world coordinate system and the camera coordinate system, the rotation and translation matrix of the camera is calculated. The specific process of step four is as follows: After obtaining the two-dimensional projected coordinates of eight key points using a regression network targeting the two-dimensional positions of key points in space, the pose is solved using the stochastic consistency perspective n-point method. The stochastic consistency perspective n-point method consists of a stochastic consistency method and a perspective n-point method. The data consists of "interior points," and these "interior point" data can be interpreted using mathematical model parameters. The process is as follows: (1) Determine the minimum sample set n required for mathematical model calculation and randomly select M minimum sample sets from all sample data P, and use these M minimum sample sets to calculate the parameters of the mathematical model respectively; (2) Calculate the error with these models using the remaining data, and identify the data as an "interior point" of the current mathematical model when the error is less than a given threshold; (3) Count the number of "interior points" under M mathematical models. The model with the most "interior points" is the correct mathematical model. Then, use the least squares method to obtain the accurate mathematical model using all the "interior point" data under the model. (4) If the corresponding set of "interior points" is not found after a certain number of samplings, the algorithm is considered to have failed; otherwise, the result obtained in step 3 shall be taken as the final result. Four interior points are selected from the key points described in step two as control points. The coordinates of these points in the target body coordinate system are known, and the two-dimensional positions of these points in the image have been obtained. All 3D points are then represented as linear combinations of these four control points; each 3D point can be represented by four control points and a set of weights, which reflect the relative position between the point and the control points, as shown in the following formula, where the 3D coordinates P of a series of 8 key points in the body coordinate system are already known. i w i = 1…8 in, Let α be the three-dimensional coordinates of the control point in the world system. ij The control coefficients are constructed by using the camera model and the representations of the 3D points obtained in the previous steps to create a linear equation system P for the positions of the control points in the camera coordinate system. i w The unknowns in the system are the coordinates of the control points and the intrinsic parameters of the camera. By solving the linear system, the positions of the control points in the camera coordinate system are obtained; this step usually involves matrix operations, and the optimal solution is found using singular value decomposition (SVD). The positions of the control points in the camera coordinate system were obtained. By comparing their positions in the world coordinate system and the camera coordinate system, the rotation and translation matrices of the camera were calculated.

2. The method for estimating the relative pose of multi-scale spatial targets in complex environments based on deep learning according to claim 1, characterized in that, In step one, various disturbances that may be encountered in the space environment are simulated, including changes in lighting, shadows, reflections, and atmospheric disturbances, to enhance the model's generalization ability and robustness.

3. The method for estimating the relative pose of multi-scale spatial targets in complex environments based on deep learning according to claim 2, characterized in that, In the rendering engine software, camera parameters and 3D model parameters of the spatial target are set. A 2D image (pose sample set) is generated by imaging the 3D model of the spatial target using the camera. The camera parameters include the distance S from the camera to the spatial target, the camera's intrinsic field of view θ, and the width and height of the spatial target image. The 3D model parameters include the position (x, y, z) of the spatial target 3D model, the three-axis attitude angles (α, θ, ψ) of the spatial target 3D model, and its rotational angular velocity.

4. The method for estimating the relative pose of multi-scale spatial targets in complex environments based on deep learning according to claim 1, characterized in that, The specific implementation process of step two is as follows: (1) The image is entered into the backbone network for downsampling processing. First, the spatial target images from the aforementioned pose sample set are input into the backbone neural network for downsampling. The backbone network output features are divided into five stages, each representing a feature vector at a different scale. The backbone network structure consists of five main stages, each composed of a series of convolutional layers and max-pooling layers, used to extract and process image features at different levels. In the first stage, the network begins preliminary feature extraction and downsampling of the input image using a 7x7 convolutional kernel (2x2 stride, 64 channels) and a 3x3 max-pooling layer (2x2 stride). Next, the second stage consists of a convolutional block (conv_blo The first stage consists of one convolutional block and two identity blocks, with all blocks configured with 64, 64, and 256 channels to further deepen feature understanding. The second stage includes one convolutional block and four identity blocks, with their channel counts increased to 128, 128, and 512 to capture more complex features. In the third stage, the network explores features more deeply through one convolutional block and eight identity blocks (with channel counts of 256, 256, and 1024). Finally, the fifth stage consists of one convolutional block and two identity blocks, with the largest number of channels, reaching 512, 512, and 2048, to complete the in-depth analysis and understanding of image features. (2) Upsampling of feature vectors in multi-scale feature networks After downsampling by the backbone network, the resulting feature vectors are then fed into the FPN (Feature Pyramid Network) for upsampling. The FPN uses a bottom-up and lateral connection structure to construct feature maps of different scales in order to capture information from coarse to fine in the image. In this process, the lower-level high-resolution feature maps are upsampled and fused with the feature maps of the upper level, gradually achieving feature refinement and scale unification. (3) Multi-scale feature fusion processing of randomized vectors After completing the upsampling of the feature vectors, a randomized fusion strategy is then used to perform multi-scale feature fusion of the randomized vectors to enhance the generalization ability and robustness of the model. By randomizing the feature maps at different scales and then combining the processed feature maps, the diversity and variability of spatial targets are captured to ensure the adaptability of different spatial targets and provide richer and more accurate feature information for subsequent key point location prediction. (4) Regression generates two-dimensional location prediction of key points The prediction of the two-dimensional position of key points of spatial targets is performed by regression using feature maps that have undergone multi-scale feature fusion processing; A regression network is constructed, and through training, it learns the mapping relationship from feature maps to the two-dimensional positions of key points; In the final stage of the network, fully connected layers or convolutional layers are used to directly perform regression prediction on the two-dimensional position of each key point; By meticulously adjusting the network parameters and structure, high-precision prediction of the location of key points of spatial targets can be achieved.

5. The method for estimating the relative pose of multi-scale spatial targets in complex environments based on deep learning according to claim 4, characterized in that, The specific process of multi-scale feature fusion processing of randomized vectors is as follows: Pixel information of the region where the spatial target is located participates in the prediction with certain weights, s k The feature map sizes at different network layers are selected based on the object size distribution in the target dataset. S is the size of the longer side of the target's two-dimensional bounding box in the original image, and N is the weight of the k-th layer. k satisfy in By utilizing information from all levels within the feature pyramid network, the robustness of the network to changes in target shooting distance is enhanced, thereby improving the overall performance and accuracy of target pose estimation.

6. The method for estimating the relative pose of multi-scale spatial targets in complex environments based on deep learning according to claim 1, characterized in that, Step three specifically involves: After constructing the regression network for the 2D positions of key points of spatial targets, the pose sample set obtained in step one is input into the network for training. The model with the smallest loss function is selected as the final training model. Before training, the loss function of the network needs to be designed. For a 3D point p i According to its projection process: make The true two-dimensional position of the projection point The epipolar direction passing through the real point, where K is the camera intrinsic parameter matrix, is the 3D keypoint p corresponding to the camera ray. i In the camera coordinate system, where R is the ground-truth rotation matrix and T is the translation vector; the reprojection error is calculated and mapped to 3D space to obtain: in It is related to camera rays The matrix of 3D points from orthogonal projection; finally, the loss is:

7. A multi-scale spatial target relative pose estimation system based on deep learning in complex environments, characterized in that: The system has a program module corresponding to the steps of any one of the claims 1-6 above, and executes the steps of the deep learning method and system for multi-scale spatial target relative pose estimation in complex environments based on deep learning when it is run.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program configured to, when invoked by a processor, implement the steps of the deep learning-based method for estimating the relative pose of multi-scale spatial targets in complex environments as described in any one of claims 1-6.

Images