Multi-scale space target relative pose estimation method and system in complex environment based on deep learning
Through the multi-scale spatial target relative pose estimation method based on deep learning, combined with the rendering engine and the random consistency perspective n-point method, the problems of insufficient pose estimation accuracy and difficulty in scale recognition under complex optical conditions are solved, and the problem of insufficient pose estimation accuracy and difficulty in scale recognition under traditional optical conditions is realized. High-precision six-degree-of-freedom pose estimation of spatial targets improves estimation accuracy and robustness.
Patent Information
- Application Number
- CN202411007553.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-07-25
AI Technical Summary
In a spatial environment, traditional visual navigation technology is difficult to achieve high-precision spatial target pose estimation under complex optical conditions, and traditional methods have large differences in identification effects when dealing with targets of different distances and scales.
Using a multi-scale spatial target relative pose estimation method based on deep learning, a three-dimensional model sample set is established through the rendering engine software, a multi-layer neural network is used to extract image features, and combined with the random consistency perspective n-point method to achieve six freedoms for spatial targets. High-precision estimation of degree poses.
Under high dynamic complex optical conditions, the accuracy and robustness of pose estimation are improved, different distance and scale recognition problems are overcome, high-precision six-degree-of-free pose estimation of spatial targets is achieved, and the generalization of the model is enhanced Ability and adaptability.
Smart Images

Figure CN119006591A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of space target pose estimation, and specifically relates to a method and system for estimating the relative pose of multi-scale space targets in a complex environment based on deep learning. Background Art
[0002] With the rapid development of aerospace technology, the complexity and diversity of space missions are increasing, and the demand for precise control of spacecraft is also increasing. These tasks involve autonomous rendezvous and docking of spacecraft, removal of space debris, on-orbit maintenance of space equipment, and many other aspects. They all require accurate estimation of the pose (position and attitude) of space targets. The accuracy of pose estimation directly affects the execution efficiency and quality of the mission, and thus becomes one of the key technologies for the success of aerospace missions. In the pose estimation technology of space targets, traditional methods such as microwave radar and GPS technology are suitable for positioning of long-distance targets, while laser radar and visual navigation technology are more suitable for close-range observation. Although laser radar technology has high accuracy, it is costly; while visual navigation technology has low cost, it is easily affected by lighting conditions. In this context, monocular vision measurement technology has become the preferred technology for pose measurement of close-range targets in space due to its advantages such as small sensor size, low energy consumption, simple structure and low cost.
[0003] In recent years, the rapid development of deep learning technology has brought new opportunities for the pose estimation of space targets. Deep learning technology, especially convolutional neural networks (CNNs), recurrent neural networks (RNNs) and graph neural networks (GNNs), has shown excellent performance in image recognition, target tracking and scene understanding. These technologies can automatically learn useful features from complex and high-dimensional data, greatly improving the accuracy and efficiency of pose estimation. In particular, by training a large number of images of spatial scenes with deep learning models, the model can learn the complex features and pose change rules of spatial targets, thereby accurately predicting the pose of unknown spatial targets. In addition, through a two-stage method, the prediction features of the spatial target model are combined with prior information to further improve the accuracy of pose estimation. With the introduction of deep learning technology, not only the problems of drastic changes in illumination encountered by traditional visual navigation technology in the pose estimation of spatial targets are solved, but also new possibilities are provided for improving the accuracy and efficiency of pose estimation of spatial targets through its powerful learning and generalization capabilities. With the continuous advancement and optimization of deep learning technology, it will show greater potential and value in the field of pose estimation of spatial targets in the future.
[0004] The prior art with document number CN113034581A discloses a method for relative pose estimation of space targets based on deep learning, which includes the following steps: a. constructing a labeled sample set using the two-dimensional projection of the three-dimensional model of the space target at different positions and postures; b. dividing the labeled sample set into a training set, a validation set and a test set, and constructing a pose estimation neural network; c. inputting the training set and the validation set into the constructed pose estimation neural network for training to obtain a pose estimation model; d. testing the test set using the trained pose estimation model to obtain the pose information of the space target of each sample in the test set. This prior art can simultaneously estimate the position and pose information of the space target through a regression model of a single image, and is suitable for target pose estimation under complex lighting conditions in space. However, the pose estimation accuracy of targets in space environments still needs to be further improved. Summary of the invention
[0005] The technical problems to be solved by the present invention are:
[0006] In order to cope with the complex optical conditions of space targets in highly dynamic space environments and the need for rapid pose calculation and to improve the pose estimation accuracy of targets in space environments, the present invention proposes a method and system for relative pose estimation of multi-scale space targets in complex environments based on deep learning (deep learning network algorithm).
[0007] The technical concept of the present invention is: after using rendering engine software (such as Unity, UrealEngine, OpenGL software or other existing software with this function, the above software is selected for trial use for scientific research purposes) to establish a sample set, a multi-layer neural network is used to extract image features, and the image plane projection point position of the predefined three-dimensional point is regressed to obtain the predicted two-dimensional position of the three-dimensional point, and then the random consistency perspective n-point method is used to solve the target motion, thereby realizing the six-degree-of-freedom posture estimation of the spatial target.
[0008] The technical solution adopted by the present invention to solve the above technical problems is:
[0009] A method for relative pose estimation of multi-scale spatial targets in a complex environment based on deep learning, the implementation process of the method is as follows:
[0010] Step 1: According to the virtual simulation of the three-dimensional model of the space target at different positions and postures, obtain the pose sample set:
[0011] Import the open source 3D model of the space target into the rendering engine software (such as Unity, UrealEngine, OpenGL software), simulate the different spatial positions and postures of the space target, and use the set internal reference K camera to shoot;
[0012] The pose sample set includes pose change data of the space target in the form of a pose transformation matrix R and a position translation vector T; the sample set includes image data when the space target is observed from multiple angles and distances, covering a wide range of lighting conditions and environmental backgrounds;
[0013] Step 2: Construct a two-dimensional position regression network for target key points:
[0014] Construct a regression network for the two-dimensional projection of the key points of the spatial target. The network sample dataset image (from the dataset image in step 1) is input into the network with Darknet53 as the backbone network. Then the downsampled image features are passed to the FPN feature pyramid network, followed by upsampling, and the two-dimensional projection of the key points is regressed. Finally, the predicted two-dimensional position of the key points is obtained through feature fusion.
[0015] Step 3: Input the regression network according to the pose sample set to obtain the target training model:
[0016] After building a regression network for the two-dimensional position of key points of the space target, the network is trained according to the pose sample set to obtain a training model that can accurately predict the target pose. The key to the training setting of this process is to ensure that the model can learn various possible poses of the space target from the pose sample set, and then accurately predict the unknown pose.
[0017] Step 4: Solve the pose using the random consistency perspective n-point method
[0018] After obtaining the two-dimensional projection of the key points, combined with the coordinates of the key points in the target body coordinate system, the control points are first selected, and the other points are expressed as linear combinations of these control points;
[0019] Then proceed as follows: use the random consistent perspective n-point method to obtain the relative position and posture between the target body coordinate system and the camera coordinate system, then construct a linear equation about the position of the control point in the camera coordinate system, use singular value decomposition (SVD) to find the optimal solution, and obtain the position of the control point in the camera coordinate system. By comparing the positions of the control points in the world coordinate system and the camera coordinate system, calculate the rotation and translation matrix of the camera (the relative position and posture of the spatial target refers to the relative position and posture, which is manifested in the posture rotation matrix and translation matrix).
[0020] Furthermore, in step one, various interferences that may be encountered in the space environment are simulated, including illumination changes, shadows, reflections, and atmospheric disturbances, to enhance the generalization ability and robustness of the model.
[0021] Furthermore, the camera parameters and the three-dimensional model parameters of the space target are set in the rendering engine software, and the three-dimensional model of the space target is imaged by the camera to generate a two-dimensional image (pose sample set); wherein the camera parameters include the camera distance S of the space target, the camera internal reference field angle range θ, and the width and height of the space target image; the three-dimensional model parameters of the space target include the three-dimensional model position (x, y, z) of the space target and the three-axis attitude angle (α, θ, ψ) of the three-axis model of the space target and its rotation angular velocity
[0022] Furthermore, the specific implementation process of step 2 is:
[0023] (1) The image enters the backbone network for downsampling
[0024] First, the spatial target image in the aforementioned pose sample set is input into the backbone neural network for downsampling. The output features of the backbone network are divided into five stages, representing feature vectors of five different scales. The backbone network structure is divided into five main stages, each of which is composed of a series of convolutional layers and maximum pooling layers to extract and process image features at different levels. In the first stage, the network starts to perform preliminary feature extraction and downsampling on the input image through a 7x7 convolution kernel (with a step size of 2x2 and 64 channels) and a 3x3 maximum pooling layer (with a step size of 2x2). Next, the second stage consists of a convolution block (conv_block The first stage consists of a convolution block and two identity blocks (identity_block), where the number of channels of all blocks is configured as 64, 64, 256 to further deepen the feature understanding; the third stage includes a convolution block and four identity blocks, and their number of channels is increased to 128, 128, 512 to capture more complex features; in the fourth stage, the network conducts a deeper feature exploration through a convolution block and eight identity blocks (with the number of channels being 256, 256, 1024); finally, the fifth stage consists of a convolution block and two identity blocks, with the largest number of channels, reaching 512, 512, 2048, to complete the in-depth analysis and understanding of image features;
[0025] (2) Upsampling of feature vectors in multi-scale feature networks
[0026] After the downsampling process of the backbone network, the obtained feature vector is then sent to the FPN (Feature Pyramid Network) for upsampling. FPN uses a bottom-up and lateral connection structure to construct feature maps of different scales to capture information from coarse to fine in the image. In this process, the high-resolution feature map of the lower level is upsampled and fused with the feature map of the previous level to gradually achieve feature refinement and scale unification.
[0027] (3) Randomized vector multi-scale feature fusion processing
[0028] After completing the upsampling of the feature vector, a randomized fusion strategy is used to perform multi-scale feature fusion processing of the randomized vector to enhance the generalization and robustness of the model. By performing random processing on feature maps of different scales and then synthesizing the processed feature maps, the diversity and variability of spatial targets are captured to ensure the adaptability of different spatial targets, providing richer and more accurate feature information for subsequent key point position prediction.
[0029] (4) Regression generates predictions of the two-dimensional positions of key points
[0030] To predict the two-dimensional position of key points of spatial targets, regression is performed using feature maps that have been processed with multi-scale feature fusion. A regression network is constructed to learn the mapping relationship from feature maps to the two-dimensional positions of key points through training. In the final stage of the network, the fully connected layer or convolutional layer is used to directly regress and predict the two-dimensional position of each key point. By carefully adjusting the network parameters and structure, high-precision prediction of the key point positions of spatial targets can be achieved.
[0031] Furthermore, the detailed training settings of step 3 are:
[0032] (1) Loss function selection: Use a loss function suitable for pose regression tasks, such as mean square error (MSE) or a customized loss function that is more suitable for specific tasks. To accurately measure the difference between the predicted pose and the true pose;
[0033] (2) Optimizer selection: Select the efficient optimizer Adam and adjust the appropriate learning rate to ensure the stability and convergence speed of the training process; at the same time, reasonably set the batch size and training rounds (Epoch) to balance the training time and model performance;
[0034] (2) Early Stopping: Monitor the performance on the validation set and stop training when the model performance stops improving after a certain number of rounds to avoid overfitting. After stopping training, jump to step 4 to solve the posture.
[0035] Furthermore, the specific process of randomized vector multi-scale feature fusion processing is as follows:
[0036] The pixel information of the area where the spatial target is located participates in the prediction with a certain weight, s k is the size of the feature maps at different levels of the network, which is selected according to the object size distribution in the target dataset. S is the size of the longer side of the target two-dimensional envelope in the original image. The weight N of the kth layer is k satisfy
[0037]
[0038] in
[0039]
[0040] By utilizing all levels of information in the feature pyramid network, the network's robustness to changes in target shooting distance is enhanced, thereby improving the overall performance and accuracy of target pose estimation.
[0041] Furthermore, step three is specifically as follows:
[0042] After constructing the 2D point regression network, the pose sample set obtained in step 1 is input into the network for training, and the model with the smallest loss function is selected as the final training. Before training, the loss function of the network needs to be designed. For the 3D point p i , according to its projection process:
[0043]
[0044] make
[0045]
[0046]
[0047] The real two-dimensional position of the projection point is the polar direction passing through the real point, K is the camera intrinsic parameter matrix, and the three-dimensional key point p corresponding to the camera ray i Expressed in the camera coordinate system, where R is the ground-truth rotation matrix and T is the translation vector; the reprojection error is mapped to 3D space by calculation:
[0048]
[0049] in
[0050]
[0051] is the camera ray The matrix of the orthogonal projection of the 3D point, finally, takes the loss as:
[0052]
[0053] Further, the specific process of step 4 is:
[0054] After using the two-dimensional point regression network to obtain the two-dimensional projection coordinates of eight key points, the random consistency perspective n-point method is used to solve the posture. The random consistency perspective n-point method consists of the random consistency method and the perspective n-point method. The random consistency method contains three assumptions: the data consists of "inside points" and these "inside point" data can be explained by mathematical model parameters, and the data consists of "outside points". The process of this method is:
[0055] (1) Determine the minimum sample set n required for mathematical model calculation and randomly select M minimum sample sets from all sample data P, and use these M minimum sample sets to calculate the parameters of the mathematical model respectively;
[0056] (2) Use the remaining data to calculate the error with these models. When the error is less than a given threshold, the data is considered to be the "interior point" of the current mathematical model.
[0057] (3) Count the number of "interior points" under the M mathematical models, and the mathematical model with the largest number of "interior points" is the correct mathematical model. Then, all the "interior point" data under the model are used to obtain the accurate mathematical model using the least squares method;
[0058] (4) If the corresponding "interior point" set is still not found after a certain number of sampling times, the algorithm is considered to have failed. Otherwise, the result obtained in the third step is taken as the final result;
[0059] Select four of the eight corner points regressed in step (2) as control points. The coordinates of these points in the target body coordinate system are known, and the two-dimensional positions of these points in the image have been obtained.
[0060] Then all 3D points are represented as linear combinations of these four control points; each 3D point can be represented by four control points and a set of weights, which reflect the relative position between the point and the control points, as shown in the following formula, where the 3D coordinates P of a series of 8 key points in the body coordinate system are known i w ,i=1…8,
[0061]
[0062] in, is the three-dimensional coordinate of the control point in the world system, α ij To control the coefficients, all points are represented by the camera model and the three-dimensional points obtained in the previous step, and a linear equation system P about the position of the control points in the camera coordinate system is constructed. i w ; The unknowns in the system are the coordinates of the control points and the intrinsic parameters of the camera;
[0063] By solving the linear system, the position of the control points in the camera coordinate system is obtained; this step usually involves matrix operations, and singular value decomposition (SVD) is used to find the optimal solution;
[0064] The positions of the control points in the camera coordinate system are obtained, and the rotation and translation matrices of the camera are calculated by comparing their positions in the world coordinate system and the camera coordinate system.
[0065] A system for estimating relative pose of multi-scale space targets in a complex environment based on deep learning. The system has a program module corresponding to the steps of the method for estimating relative pose of multi-scale space targets in a complex environment based on deep learning, and executes the steps of the method for estimating relative pose of multi-scale space targets in a complex environment based on deep learning during operation.
[0066] A computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of a method for relative pose estimation of multi-scale spatial targets in a complex environment based on deep learning when called by a processor.
[0067] The present invention has the following beneficial technical effects:
[0068] The deep learning network algorithm of the present invention provides an efficient and accurate solution for space target pose estimation. By adopting an encoder-decoder structure, this algorithm can deeply mine the rich information of image data and achieve high-precision estimation of the six-degree-of-freedom pose of space targets. The effects and advantages of the invention are described in detail below:
[0069] (1) Stability under highly dynamic and complex optical conditions
[0070] The optical conditions in the space environment are complex and changeable, including direct sunlight, reflected light from the earth and interference from other artificial light sources. These conditions pose a great challenge to the estimation of the position and posture of space targets. The algorithm of the present invention effectively extracts the key features of the target under different lighting conditions by fusing the regression results of each layer of the FPN network, enhances the model's adaptability to complex optical environments, and ensures accurate estimation of the target's position and posture in highly dynamically changing space environments.
[0071] (2) Overcoming the recognition problem at different distances and scales
[0072] Traditional spatial target pose estimation methods have great differences in effect when dealing with targets of different distances and scales, especially the recognition and positioning of distant targets are more difficult. The present invention integrates the feature layer of the FPN network, allowing the model to simultaneously utilize image information of different scales, thereby improving the recognition accuracy of targets of various distances and scales, and ensuring the accuracy and robustness of pose estimation.
[0073] (3) Accurate 3D point pose calculation
[0074] After extracting image features using a multi-layer neural network, the algorithm performs regression analysis on the position of the image plane projection point for the predefined three-dimensional points, and then uses the random consistency perspective n-point method to solve the target motion, achieving high-precision six-degree-of-freedom pose estimation of the space target. Compared with traditional solution technologies, this method can more accurately reflect the actual position and posture of the space target, improving the accuracy and efficiency of pose estimation.
[0075] (4) Innovation of error assessment methods
[0076] The present invention adopts an error evaluation method that reprojects two-dimensional points into three-dimensional space. This method can effectively overcome the problem that traditional error evaluation is affected by target distance. Through this method, the algorithm can more accurately evaluate the error of pose estimation, thereby further optimizing the model performance and ensuring the high reliability of pose estimation results.
[0077] (5) Test performance
[0078] In the final test set, the algorithm of the present invention showed excellent performance, with the average value of the three-axis Euler angle error being only 1.92° and the maximum value not exceeding 5°, showing the high accuracy and stability of the model in estimating the position and posture of space targets. These test results fully demonstrate the efficiency and reliability of the algorithm of the present invention in practical applications, and provide solid technical support for the execution of future space missions. BRIEF DESCRIPTION OF THE DRAWINGS
[0079] Figure 1 is the pose estimation network flowchart, Figure 2 This is a schematic diagram of a satellite in orbit. Figure 3 Figure 4 is a flowchart of the pose estimation network structure, Figure 5 is a diagram of the Darknet53 structure, and Figure 6 is a diagram of the multi-scale feature pyramid network structure. Figure 7 is the target pose estimation graph, Figure 8 9 is a set of error tracking diagrams, in which: (a) is the Z-axis error tracking diagram, (b) is the Y-axis error tracking diagram, and (c) is the Z-axis error tracking diagram. DETAILED DESCRIPTION
[0080] Combined with Figure 1 -9 The implementation of the present invention is described as follows:
[0081] The present application embodiment provides a deep learning method and system for estimating relative pose of multi-scale spatial targets in a complex environment based on deep learning, and the method steps include:
[0082] Step 1: Obtain a set of pose samples based on virtual simulation of the three-dimensional model of the space target at different positions and postures
[0083] This step uses the virtual technology provided by the Unity simulation software to build a three-dimensional model of the space target and simulate the space target in different spatial positions and postures. By rendering these three-dimensional models with high realism and shooting with a camera with set internal reference K, the pose sample set contains rich space target pose change data, which exists in the form of pose transformation matrix R and position translation vector T. The sample set contains image data when observing the space target from multiple angles and distances, covering a wide range of lighting conditions and environmental backgrounds. In addition, this step also simulates various interferences that may be encountered in the space environment, such as lighting changes, shadows, reflections, and atmospheric disturbances, to enhance the generalization ability and robustness of the model.
[0084] The shooting software also performs virtual simulation rendering based on Unity, sets camera parameters and space target three-dimensional model parameters in Unity, and generates a two-dimensional image by imaging the space target three-dimensional model through the camera; the camera parameters include the camera distance S for shooting the space target, the camera internal reference field of view angle range θ, and the width and height of the space target image; the space target three-dimensional model parameters include the space target three-dimensional model position (x, y, z) and the three-axis attitude angle (α, θ, ψ) of the space target three-axis model and its rotation angular velocity (α&, θ&, ψ&).
[0085] Step 2: Construct a two-dimensional position regression network for target key points
[0086] The present invention constructs a regression network for the two-dimensional projection of the key points of spatial targets based on deep learning. The network sample data set image is input into a network with Darknet53 as the backbone network, and then the downsampled image features are passed to the FPN feature pyramid network, followed by upsampling. Finally, the two-dimensional projection of the key points is regressed after feature fusion to obtain the predicted two-dimensional position of the key points.
[0087] (1) The image enters the backbone network for downsampling
[0088] First, the spatial target images in the above sample set are input into the backbone neural network for downsampling. The output features of the backbone network are divided into five stages, representing feature vectors of five different scales. The backbone network structure is divided into five main stages, each of which is composed of a series of convolutional layers and maximum pooling layers, specially designed to extract and process image features at different levels. In the first stage, the network starts preliminary feature extraction and downsampling of the input image through a 7x7 convolution kernel (with a step size of 2x2 and a channel number of 64) and a 3x3 maximum pooling layer (with a step size of 2x2). Next, the second stage consists of a convolution block (conv_block) and two identity blocks (identity_block), where the number of channels of all blocks is configured as 64, 64, and 256 to further deepen the feature understanding. The third stage includes a convolution block and four identity blocks, whose channel numbers are increased to 128, 128, and 512 to capture more complex features. In the fourth stage, the network conducts a deeper feature exploration through a convolutional block and eight identity blocks with the number of channels being 256, 256, and 1024. Finally, the fifth stage consists of a convolutional block and two identity blocks with the largest number of channels, reaching 512, 512, and 2048, to complete the in-depth analysis and understanding of image features.
[0089] (2) Upsampling of feature vectors in multi-scale feature networks
[0090] According to one aspect of the present invention, after the downsampling process of the backbone network, the obtained feature vector is then sent to the FPN (Feature Pyramid Network) for upsampling. FPN uses a bottom-up and lateral connection structure to construct feature maps of different scales to capture information from coarse to fine in the image. In this process, the high-resolution feature map of the lower level is upsampled and fused with the feature map of the upper level, gradually realizing the refinement of features and the unification of scales.
[0091] (3) Randomized vector multi-scale feature fusion processing
[0092] In the present invention, after completing the upsampling process of the feature vector, the next step is the multi-scale feature fusion process of the randomized vector. This step adopts a randomized fusion strategy to enhance the generalization ability and robustness of the model. By performing random processing on feature maps of different scales and then combining these processed feature maps, the diversity and variability of spatial targets can be more effectively captured. Such processing can ensure the adaptability of different spatial targets and also provide more abundant and accurate feature information for subsequent key point position prediction.
[0093] (4) Regression generates predictions of the two-dimensional positions of key points
[0094] The prediction of the two-dimensional position of the key points of the space target in the present invention is to regress the feature map after multi-scale feature fusion processing. The regression network constructed here learns the mapping relationship from the feature map to the two-dimensional position of the key points through training. In the final stage of the network, the two-dimensional position of each key point is directly regressed and predicted using a fully connected layer or a convolutional layer. By carefully adjusting the network parameters and structure, high-precision prediction of the key point position of the space target can be achieved.
[0095] Step 3: Input the regression network according to the pose sample set to obtain the target training model
[0096] The present invention constructs a regression network for the two-dimensional position of the key points of the space target, and then trains the network according to the pose sample set to obtain a training model that can accurately predict the target pose. The key to the training setting of this process is to ensure that the model can learn various possible poses of the space target from the pose sample set, and then accurately predict the unknown pose. The following is a detailed training setting for this step as an example:
[0097] (1) Loss function selection: Use a loss function suitable for the pose regression task, such as the mean square error (MSE) or a customized loss function that is more suitable for the specific task, to accurately measure the difference between the predicted pose and the true pose.
[0098] (2) Optimizer selection: Select the efficient optimizer Adam and adjust the appropriate learning rate to ensure the stability and convergence speed of the training process. At the same time, reasonably set the batch size and training rounds (Epochs) to balance the training time and model performance.
[0099] (2) Early Stopping: Monitor the performance on the validation set and stop training when the model performance stops improving after a certain number of rounds to avoid overfitting.
[0100] Step 4: Solve the pose using the random consistency perspective n-point method
[0101] After obtaining the two-dimensional projection of the key points, combined with the coordinates of the key points in the target body coordinate system, first select the control points, express the other points as the linear combination of these control points, and then use the random consistent perspective n-point method to obtain the relative pose between the target body coordinate system and the camera coordinate system. Then construct a linear equation about the position of the control points in the camera coordinate system, use the singular value decomposition (SVD) to find the optimal solution, and get the position of the control points in the camera coordinate system. By comparing their positions in the world coordinate system and the camera coordinate system, calculate the rotation and translation matrices of the camera.
[0102] Example:
[0103] The following is based on the attached Figure 1-9 describes the present invention in detail.
[0104] Flowchart of the relative pose estimation method and system construction method for space targets based on deep learning, such as Figure 1 shown.
[0105] Step 1: Obtain a set of pose samples based on virtual simulation of the three-dimensional model of the space target at different positions and postures
[0106] Establishing a 3D model of space targets on orbit Figure 2 As shown, the relative pose estimation method of space targets based on deep learning of the present invention firstly marks the key points of the space targets of the three-dimensional model with envelopes, and specifically marks the envelopes of the selected satellite bodies. Figure 3 As shown, the pose data sample dataset is output through the pose conversion relationship between the envelope position and the coordinate system between the camera and the position translation vector. Then the labeled sample set is divided into training set, validation set and test set, and the pose estimation neural network is constructed. The ratio of training set, validation set and test set in the dataset is shown in Table 1 below;
[0107] Table 1. Space target relative pose estimation image dataset
[0108] Training set Test Set Validation set 35,000 2500 sheets 2500 sheets
[0109] Step 2: Construct a two-dimensional position regression network for target key points
[0110] The present invention constructs a regression network for the two-dimensional projection of the key points of spatial targets based on deep learning. The network sample data set image is input into a network with Darknet53 as the backbone network, and then the downsampled image features are passed to the FPN feature pyramid network. After that, upsampling processing is performed, and the two-dimensional projection of the key points is regressed. Finally, the predicted two-dimensional position of the key points is obtained through feature fusion.
[0111] (1) The image enters the backbone network for downsampling
[0112] The present invention designs a deep learning model based on the Darknet53 backbone network to extract key features from satellite images and predict the two-dimensional position of key points of satellites. The specific backbone network structure diagram is shown in Figure 5.
[0113] First, the spatial target images in the sample set are input into the backbone neural network for downsampling. The output features of the backbone network are divided into five stages, representing feature vectors of five different scales. The backbone network structure is divided into five main stages, each of which is composed of a series of convolutional layers and maximum pooling layers, which are specially designed to extract and process image features at different levels.
[0114] ① Stage 1 operation: The image first enters the backbone network and undergoes downsampling. ; Convolution layer: A 7x7 convolution kernel is used for convolution operation, the step size is set to 2x2, and the number of channels is 64. This design helps to initially extract the basic features of the image. ; Max pooling layer: Following the convolution layer, a 3x3 max pooling layer is used for further downsampling, and the step size is also set to 2x2, which helps to reduce the feature dimension while retaining important feature information.
[0115] ②Stage 2: It consists of one convolution block and two identity blocks. Channel number configuration: In this stage, the number of channels of the convolution block and the identity block are configured to 64, 64, and 256 respectively. This design deepens the network's understanding of image features and improves the ability to represent features.
[0116] ③Stage 3: It consists of one convolution block and four identity blocks. Channel number increase: The number of channels in this stage is increased to 128, 128, and 512 to capture more complex and detailed features. With more identity blocks, the network can learn deeper feature representations.
[0117] ④Stage 4: It consists of one convolution block and eight identity blocks. Number of channels: The number of channels is further increased to 256, 256, and 1024. This stage aims to explore image features through a deeper network structure and accurately capture the details of the target.
[0118] ⑤Stage 5: One convolution block and two identity blocks constitute the final stage. Maximum number of channels: In this stage, the number of channels reaches the maximum, set to 512, 512, 2048. This design completes the in-depth analysis and understanding of image features and provides sufficient feature support for the precise positioning of key points.
[0119] (2) Upsampling of feature vectors in multi-scale feature networks
[0120] According to one aspect of the present invention, after the downsampling process of the backbone network, the obtained feature vector is then sent to the FPN (feature pyramid network) for upsampling. The structure of the multi-feature pyramid network FPN is shown in Figure 6. First is the bottom-up path structure. The bottom-up path gradually reduces the spatial size of the feature map through a series of convolutional layers and pooling layers, while increasing the depth of the feature map, thereby capturing features from shallow to deep layers. Then there is the top-down path. The top-down path starts from the deepest feature map output by the bottom-up path, gradually enlarges the feature map, and adapts to the size of the shallower feature map.
[0121] (3) Randomized vector multi-scale feature fusion processing
[0122] In the present invention, feature maps of different scales are weighted and fused, and then these processed feature maps are integrated to more effectively capture the diversity and variability of spatial targets. This method allows the pixel information in the target area to participate in the prediction with a certain weight. k is the size of the feature maps at different levels of the network, which is selected according to the object size distribution in the target dataset. S is the size of the longer side of the target 2D envelope in the original image. The weight N of the kth layer k satisfy
[0123]
[0124] in
[0125]
[0126] By utilizing all levels of information in the feature pyramid network, the network's robustness to changes in target shooting distance is enhanced, thereby improving the overall performance and accuracy of target pose estimation.
[0127] (4) Regression generates predictions of the two-dimensional positions of key points
[0128] The prediction of the two-dimensional position of the key points of the space target in the present invention is to regress the feature map after multi-scale feature fusion processing. The regression network constructed here learns the mapping relationship from the feature map to the two-dimensional position of the key points through training. In the final stage of the network, the two-dimensional position of each key point is directly regressed and predicted using a fully connected layer or a convolutional layer. By carefully adjusting the network parameters and structure, high-precision prediction of the key point position of the space target can be achieved.
[0129] Step 3: Input the regression network according to the pose sample set to obtain the target training model
[0130] After building the 2D point regression network, input the pose sample set obtained in step 1 into the network for training, train it, and select the model with the smallest loss function as the final training. Before training, the loss function of the network needs to be designed. For the 3D point p i , according to its projection process:
[0131]
[0132] make
[0133]
[0134] The real two-dimensional position of the projection point is the polar direction passing through the real point, K is the camera intrinsic parameter matrix, and the three-dimensional key point p corresponding to the camera ray iExpressed in the camera coordinate system, where R is the ground-truth rotation matrix and T is the translation vector; the reprojection error is mapped to 3D space by calculation:
[0135]
[0136] in
[0137]
[0138] It is a ray with the camera The matrix of the 3D point of the orthogonal projection, finally, the loss is
[0139]
[0140] The hyper parameters of the pose estimation network are set in the present invention as shown in Table 2 below:
[0141] Table 3 Corner position regression network hyperparameters
[0142]
[0143] Step 4: Solve the pose using the random consistency perspective n-point method
[0144] After using the two-dimensional point regression network to obtain the two-dimensional projection coordinates of the eight key points, the random consistency perspective n-point method is used to solve the pose. The random consistency perspective n-point method consists of the random consistency method and the perspective n-point method. The random consistency method contains three assumptions: the data consists of "inside points" and these "inside point" data can be explained by mathematical model parameters, the data consists of "outside points" and these "outside point" data cannot use mathematical models, and the data contains noisy data. The process of this method is:
[0145] (1) Determine the minimum sample set n required for mathematical model calculation and randomly select M minimum sample sets from all sample data P, and use these M minimum sample sets to calculate the parameters of the mathematical model respectively.
[0146] (2) Use the remaining data to calculate the error with these models. When the error is less than a given threshold, the data is identified as the "interior point" of the current mathematical model.
[0147] (3) Count the number of “interior points” under M mathematical models, and the mathematical model with the largest number of “interior points” is the correct mathematical model. Then, all the “interior point” data under the model are used to obtain the accurate mathematical model using the least squares method.
[0148] (4) If the corresponding “interior point” set is still not found after a certain number of samplings, the algorithm is considered to have failed. Otherwise, the result obtained in the third step is taken as the final result.
[0149] Four of the eight corner points regressed in step 2 are selected as control points. The coordinates of these points in the target body coordinate system are known, and the two-dimensional positions of these points in the image have been obtained.
[0150] Then all 3D points are represented as linear combinations of these four control points. This means that each 3D point can be represented by four control points and a set of weights that reflect the relative position between the point and the control points, as shown in the following formula, where the 3D coordinates P of a series of 8 key points in the body coordinate system are known. i w ,i=1…8,
[0151]
[0152] in, is the three-dimensional coordinate of the control point in the world system, α ij To control the coefficients, all points are represented by the camera model and the three-dimensional points obtained in the previous step, and a linear equation system P about the position of the control points in the camera coordinate system is constructed. i w ; The unknowns in the system are the coordinates of the control points and the intrinsic parameters of the camera;
[0153] By solving the linear system, the positions of the control points in the camera coordinate system are obtained. This step usually involves matrix operations, and singular value decomposition (SVD) is used to find the optimal solution.
[0154] The positions of the control points in the camera coordinate system are obtained, and the rotation and translation matrices of the camera are calculated by comparing their positions in the world coordinate system and the camera coordinate system.
Claims
1. A method for relative pose estimation of multi-scale spatial targets in complex environments based on deep learning, characterized in that: The implementation process of the method is: Step 1: According to the virtual simulation of the three-dimensional model of the space target at different positions and postures, obtain the pose sample set: Import the open-source 3D model of the space target into the rendering engine software, simulate the different spatial positions and postures of the space target, and use the set intrinsic reference K camera to shoot; The pose sample set includes pose change data of the space target in the form of a pose transformation matrix R and a position translation vector T; the sample set includes image data when the space target is observed from multiple angles and distances, covering a wide range of lighting conditions and environmental backgrounds; Step 2: Construct a two-dimensional position regression network for target key points: Construct a regression network for the two-dimensional projection of the key points of the spatial target. The network sample data set image is input into the network composed of Darknet53 as the backbone network, and then the downsampled image features are passed to the FPN feature pyramid network. After upsampling, the two-dimensional projection of the key points is regressed, and finally the predicted two-dimensional position of the key points is obtained through feature fusion. Step 3: Input the regression network according to the pose sample set to obtain the target training model: After building a regression network for the two-dimensional position of key points of the space target, the network is trained according to the pose sample set to obtain a training model that can accurately predict the target pose. The key to the training setting of this process is to ensure that the model can learn various possible poses of the space target from the pose sample set, and then accurately predict the unknown pose. Step 4: Solve the pose using the random consistency perspective n-point method After obtaining the two-dimensional projection of the key points, combined with the coordinates of the key points in the target body coordinate system, the control points are first selected, and the other points are expressed as linear combinations of these control points; Then proceed as follows: use the random consistent perspective n-point method to obtain the relative pose between the target body coordinate system and the camera coordinate system, then construct a linear equation about the position of the control point in the camera coordinate system, use singular value decomposition (SVD) to find the optimal solution, and obtain the position of the control point in the camera coordinate system. By comparing the positions of the control points in the world coordinate system and the camera coordinate system, calculate the rotation and translation matrix of the camera.
2. According to the method of multi-scale spatial target relative pose estimation in a complex environment based on deep learning in claim 1, it is characterized in that: In step one, various disturbances that may be encountered in the space environment are simulated, including illumination changes, shadows, reflections, and atmospheric disturbances, to enhance the generalization and robustness of the model.
3. According to the method for relative pose estimation of multi-scale space targets in complex environments based on deep learning in claim 2, it is characterized in that: The camera parameters and the three-dimensional model parameters of the space target are set in the rendering engine software, and the three-dimensional model of the space target is imaged by the camera to generate a two-dimensional image (pose sample set); the camera parameters include the camera distance S of the space target, the camera internal reference field angle range θ, and the width and height of the space target image; the three-dimensional model parameters of the space target include the position (x, y, z) of the three-dimensional model of the space target and the three-axis attitude angle (α, θ, ψ) of the three-axis model of the space target and its rotation angular velocity 4. According to the method for relative pose estimation of multi-scale spatial targets in complex environments based on deep learning in claim 1, it is characterized in that: The specific implementation process of step 2 is: (1) The image enters the backbone network for downsampling First, the spatial target image in the aforementioned pose sample set is input into the backbone neural network for downsampling. The output features of the backbone network are divided into five stages, representing feature vectors of five different scales. The backbone network structure is divided into five main stages, each of which is composed of a series of convolutional layers and maximum pooling layers to extract and process image features at different levels. In the first stage, the network starts to perform preliminary feature extraction and downsampling on the input image through a 7x7 convolution kernel (with a step size of 2x2 and 64 channels) and a 3x3 maximum pooling layer (with a step size of 2x2). Next, the second stage consists of a convolution block (conv_block The first stage consists of a convolution block and two identity blocks (identity_block), where the number of channels of all blocks is configured as 64, 64, 256 to further deepen the feature understanding; the third stage includes a convolution block and four identity blocks, and their number of channels is increased to 128, 128, 512 to capture more complex features; in the fourth stage, the network conducts a deeper feature exploration through a convolution block and eight identity blocks (with the number of channels being 256, 256, 1024); finally, the fifth stage consists of a convolution block and two identity blocks, with the largest number of channels, reaching 512, 512, 2048, to complete the in-depth analysis and understanding of image features; (2) Upsampling of feature vectors in multi-scale feature networks After the downsampling process of the backbone network, the obtained feature vector is then sent to the FPN (Feature Pyramid Network) for upsampling. FPN uses a bottom-up and lateral connection structure to construct feature maps of different scales to capture information from coarse to fine in the image. In this process, the high-resolution feature map of the lower level is upsampled and fused with the feature map of the previous level to gradually achieve feature refinement and scale unification. (3) Randomized vector multi-scale feature fusion processing After completing the upsampling of the feature vector, a randomized fusion strategy is used to perform multi-scale feature fusion processing of the randomized vector to enhance the generalization and robustness of the model. By performing random processing on feature maps of different scales and then synthesizing the processed feature maps, the diversity and variability of spatial targets are captured to ensure the adaptability of different spatial targets, providing richer and more accurate feature information for subsequent key point position prediction. (4) Regression generates predictions of the two-dimensional positions of key points The prediction of the two-dimensional position of the key points of the spatial target is performed by regression using the feature map processed by multi-scale feature fusion; Build a regression network and learn the mapping relationship from feature maps to the two-dimensional positions of key points through training; In the final stage of the network, a fully connected layer or convolutional layer is used to directly regress and predict the two-dimensional position of each key point; By carefully adjusting the network parameters and structure, high-precision prediction of the key point positions of spatial targets can be achieved.
5. According to the method of multi-scale spatial target relative pose estimation in a complex environment based on deep learning in claim 1, it is characterized in that: The detailed training settings for step 3 are: (1) Loss function selection: Use a loss function suitable for pose regression tasks, such as mean square error (MSE) or a customized loss function that is more suitable for specific tasks. To accurately measure the difference between the predicted pose and the true pose; (2) Optimizer selection: Select the efficient optimizer Adam and adjust the appropriate learning rate to ensure the stability and convergence speed of the training process; at the same time, reasonably set the batch size and training rounds (Epoch) to balance the training time and model performance; (2) Early Stopping: Monitor the performance on the validation set and stop training when the model performance stops improving after a certain number of rounds to avoid overfitting. After stopping training, jump to step 4 to solve the posture.
6. According to the method of multi-scale spatial target relative pose estimation in a complex environment based on deep learning in claim 4, it is characterized in that: The specific process of randomized vector multi-scale feature fusion processing is as follows: The pixel information of the area where the spatial target is located participates in the prediction with a certain weight, s k is the size of the feature maps at different levels of the network, which is selected according to the object size distribution in the target dataset. S is the size of the longer side of the target two-dimensional envelope in the original image. The weight N of the kth layer is k satisfy in By utilizing all levels of information in the feature pyramid network, the network's robustness to changes in target shooting distance is enhanced, thereby improving the overall performance and accuracy of target pose estimation.
7. According to the method of multi-scale spatial target relative pose estimation in a complex environment based on deep learning in claim 5, it is characterized in that: Step three is as follows: After constructing the 2D point regression network, the pose sample set obtained in step 1 is input into the network for training, and the model with the smallest loss function is selected as the final training. Before training, the loss function of the network needs to be designed. For the 3D point p i , according to its projection process: make The real two-dimensional position of the projection point is the polar direction passing through the real point, K is the camera intrinsic parameter matrix, and the three-dimensional key point p corresponding to the camera ray i Expressed in the camera coordinate system, where R is the ground-truth rotation matrix and T is the translation vector; the reprojection error is mapped to 3D space by calculation: in is the camera ray The matrix of the orthogonal projection of the 3D point, finally, takes the loss as:
8. According to the method of multi-scale spatial target relative pose estimation in a complex environment based on deep learning in claim 1, it is characterized in that: The specific process of step 4 is as follows: After using the two-dimensional point regression network to obtain the two-dimensional projection coordinates of eight key points, the random consistency perspective n-point method is used to solve the posture. The random consistency perspective n-point method consists of the random consistency method and the perspective n-point method. The random consistency method contains three assumptions: the data consists of "inside points" and these "inside point" data can be explained by mathematical model parameters, and the data consists of "outside points". The process of this method is: (1) Determine the minimum sample set n required for mathematical model calculation and randomly select M minimum sample sets from all sample data P, and use these M minimum sample sets to calculate the parameters of the mathematical model respectively; (2) Use the remaining data to calculate the error with these models. When the error is less than a given threshold, the data is considered to be the "interior point" of the current mathematical model. (3) Count the number of "interior points" under the M mathematical models, and the mathematical model with the largest number of "interior points" is the correct mathematical model. Then, all the "interior point" data under the model are used to obtain the accurate mathematical model using the least squares method; (4) If the corresponding "interior point" set is still not found after a certain number of sampling times, the algorithm is considered to have failed. Otherwise, the result obtained in the third step is taken as the final result. Select four of the eight corner points regressed in step (2) as control points. The coordinates of these points in the target body coordinate system are known, and the two-dimensional positions of these points in the image have been obtained. Then all 3D points are represented as linear combinations of these four control points; each 3D point can be represented by four control points and a set of weights, which reflect the relative position between the point and the control points, as shown in the following formula, where the 3D coordinates of a series of 8 key points in the body coordinate system are known i=1…8, in, is the three-dimensional coordinate of the control point in the world system, α ij To control the coefficients, all points are represented using the camera model and the 3D points obtained in the previous step to construct a linear equation system about the position of the control points in the camera coordinate system. The unknowns in the system are the coordinates of the control points and the intrinsic parameters of the camera; By solving the linear system, the position of the control points in the camera coordinate system is obtained; this step usually involves matrix operations, and singular value decomposition (SVD) is used to find the optimal solution; The positions of the control points in the camera coordinate system are obtained, and the rotation and translation matrices of the camera are calculated by comparing their positions in the world coordinate system and the camera coordinate system.
9. A multi-scale spatial target relative pose estimation system in a complex environment based on deep learning, characterized by: The system has a program module corresponding to the steps of any one of claims 1 to 8 above, and executes the steps in the deep learning method and system for relative pose estimation of multi-scale spatial targets in a complex environment based on deep learning when running.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the method for relative pose estimation of multi-scale spatial targets in a complex environment based on deep learning according to any one of claims 1 to 8 when called by a processor.
Citation Information
Patent Citations
Deep learning and geometric algorithm combined non-cooperative target relative pose estimation method
CN111862126A
Workpiece 6D pose estimation method based on deep learning
CN111899301A
Space target relative pose estimation method based on deep learning
CN113034581A
Three-dimensional reconstruction method and device, electronic equipment and computer storage medium
CN113178000A
Space target relative pose iterative estimation method and system based on key point weight
CN116645392A
Cited By
Three-dimensional human body posture recognition method based on photogrammetry and deep learning
CN119339443A
Spacecraft pose measurement system based on monocular vision image processing
CN119722794A
Camera repositioning method for approaching landing of fixed-wing aircraft
CN119832459A
Aerial non-cooperative target attitude estimation method and system
CN120765742A
Task processing method, model training method and system, electronic equipment, storage medium and computer program product
CN121564526A
