Spatial non-cooperative target pose estimation method based on multi-scale dense structure neural network
Through multi-scale dense structural neural network, high-precision space non-cooperative target pose estimation is achieved in complex space environments, solving the problem of traditional methods with low background complexity and low visibility, providing more accurate space tasks Posture information.
Patent Information
- Application Number
- CN202411228084.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-03
- Publication Date
- 2024-12-06
AI Technical Summary
The prior art is difficult to achieve high-precision space non-cooperation target estimation in space environments with complex backgrounds and low visibility, and traditional methods are greatly affected by lighting changes and background complexity.
Using a monocular space non-cooperative target pose estimation method based on multi-scale dense structural neural network, the end-to-end six-dimensional six-dimensional The degree of freedom pose estimation improves the accuracy of pose estimation in complex environments.
In an environment with complex background and low visibility, the accuracy of pose estimation is improved, and new ideas are provided for subsequent target capture and fault detection tasks.
Smart Images

Figure CN119085666A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of spacecraft navigation, and in particular to a method for estimating the position and posture of a space non-cooperative target. Background Art
[0002] In space confrontation and on-orbit service missions, friendly spacecraft need to perform tasks such as circling, approaching, capturing, rendezvous and docking with non-cooperative space targets. Relative attitude estimation of non-cooperative space targets is a key technology. Unlike cooperative spacecraft, researchers cannot communicate with non-cooperative space targets and it is difficult to obtain prior information, which brings great difficulties to attitude estimation technology. Pose estimation can determine the position, size and motion state of space objects in real time. Monocular cameras are widely used in space attitude estimation tasks due to their small size, low power consumption and simple hardware requirements.
[0003] Traditional monocular pose estimation methods are mainly based on image feature extraction technology, such as ORB and SIFT methods, which extract local features from images, compare them with the features obtained on the model, establish 2D-3D feature point matching, and use correspondence to solve geometric problems. However, traditional pose estimation methods require that images of non-cooperative targets in space have clear texture details, which is greatly limited in space environments with drastic lighting changes and complex backgrounds.
[0004] With the rise of artificial intelligence, deep learning, with its advantages of autonomous learning ability and strong robustness, has provided a new direction for the pose estimation technology of non-cooperative space targets. Deep learning technology can improve the robustness of image processing by learning a large amount of data, so that pose estimation can still be performed accurately in the complex environment of deep space. Autonomous space exploration can be achieved by using spacecraft equipped with advanced visual image sensors and powerful image data computing capabilities. The application of artificial intelligence technologies such as deep learning is the key driving force for promoting the development of spacecraft to a higher level of intelligence.
[0005] Therefore, the existing technology requires a method that can build an end-to-end neural network based on deep learning to realize the pose estimation of spatial non-cooperative targets in a complex space environment, so as to achieve high-precision pose estimation in an environment with complex background and low visibility. Summary of the invention
[0006] The purpose of the present invention is to solve the problem of low pose estimation accuracy of existing spatial non-cooperative target pose estimation methods in environments with complex backgrounds and low visibility, and to propose a spatial non-cooperative target pose estimation method based on a multi-scale dense structure neural network.
[0007] The specific process of the spatial non-cooperative target pose estimation method based on multi-scale dense structure neural network is as follows:
[0008] Step 1: Acquire the image of the spatial non-cooperative target, the position information of the spatial non-cooperative target, and the quaternion information of the spatial non-cooperative target;
[0009] A training set is constructed based on the spatial non-cooperative target image, the position information of the spatial non-cooperative target, and the quaternion information of the spatial non-cooperative target;
[0010] A spatial non-cooperative target image corresponds to position information of a spatial non-cooperative target and quaternion information of a spatial non-cooperative target;
[0011] The number of spatial non-cooperative target images, the number of spatial non-cooperative target position information, and the number of spatial non-cooperative target quaternion information are all N;
[0012] Step 2: Convert the quaternion information in the training set of step 1 into Euler angles, encode the Euler angles based on the soft classification idea, and obtain the true value of the probability mass function of each Euler angle;
[0013] Step 3: Construct a multi-scale dense structure pose estimation network to obtain a trained multi-scale dense structure pose estimation network;
[0014] Step 4: Input the non-cooperative target image in the tested space into the trained multi-scale dense structure pose estimation network, and the trained multi-scale dense structure pose estimation network outputs the predicted probability mass function;
[0015] Decoding the predicted probability mass function to obtain the decoded Euler angle;
[0016] The decoded Euler angle Follow step 2 to convert back to quaternion format Will Combined with the position of the spatial non-cooperative target on the X, Y, and Z axes output by the trained multi-scale dense structure pose estimation network, the six-degree-of-freedom pose information of the spatial non-cooperative target is obtained.
[0017] The beneficial effects of the present invention are:
[0018] In order to solve the above problems existing in the prior art, the present invention proposes a monocular spatial non-cooperative target pose estimation method based on a multi-scale dense neural network and spatial pose soft classification. The method takes a monocular image as input, designs a feature extraction network framework to complete image feature extraction, and end-to-end regresses the six-degree-of-freedom position and pose of the spatial non-cooperative target, which improves the pose estimation accuracy in environments with complex backgrounds and low visibility, thereby providing new ideas for subsequent on-orbit engineering tasks such as target capture and fault detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1This is a flowchart of a method for estimating six-degree-of-freedom pose of a spatial non-cooperative target based on a multi-scale dense structure neural network;
[0020] Figure 2 It is a structural diagram of the multi-scale convolution module;
[0021] Figure 3 This is a diagram of the multi-scale dense structure pose estimation network structure;
[0022] Figure 4 are satellite model images collected by a monocular visible light camera in a simulation environment, a is satellite model image 1 collected by a monocular visible light camera, b is satellite model image 2 collected by a monocular visible light camera, and c is satellite model image 3 collected by a monocular visible light camera;
[0023] Figure 5 is the visualization result of pose estimation, a is the visualization result of pose estimation Figure 1 , b is the visualization result of pose estimation Figure 2 . DETAILED DESCRIPTION
[0024] Specific implementation method 1: The specific process of the spatial non-cooperative target pose estimation method based on a multi-scale dense structure neural network in this implementation method is as follows:
[0025] Step 1: Acquire the image of the spatial non-cooperative target, the position information of the spatial non-cooperative target, and the quaternion information of the spatial non-cooperative target;
[0026] Construct a training set (with labels) based on the spatial non-cooperative target image, the position information of the spatial non-cooperative target and the quaternion information of the spatial non-cooperative target;
[0027] A spatial non-cooperative target image corresponds to position information of a spatial non-cooperative target and quaternion information of a spatial non-cooperative target;
[0028] The number of spatial non-cooperative target images, the number of spatial non-cooperative target position information, and the number of spatial non-cooperative target quaternion information are all N; N is a positive integer;
[0029] Step 2: Convert the quaternion information (labeled) in the training set of step 1 into Euler angles (labeled), encode the Euler angles based on the soft classification idea, and obtain the true value of the probability mass function of each Euler angle;
[0030] Step 3: Construct a multi-scale dense structure pose estimation network to obtain a trained multi-scale dense structure pose estimation network;
[0031] Step 4: Input the non-cooperative target image in the tested space into the trained multi-scale dense structure pose estimation network, and the trained multi-scale dense structure pose estimation network outputs the predicted probability mass function;
[0032] Decoding the predicted probability mass function to obtain the decoded Euler angle;
[0033] The decoded Euler angle Follow step 2 to convert back to quaternion format Will Combined with the position of the spatial non-cooperative target on the X, Y, and Z axes output by the trained multi-scale dense structure pose estimation network, the six-degree-of-freedom pose information of the spatial non-cooperative target is obtained.
[0034] Specific implementation method 2: This implementation method is different from specific implementation method 1 in that in step 1, the spatial non-cooperative target image, the position information of the spatial non-cooperative target and the quaternion information of the spatial non-cooperative target are obtained;
[0035] Construct a training set (with labels) based on the spatial non-cooperative target image, the position information of the spatial non-cooperative target and the quaternion information of the spatial non-cooperative target;
[0036] A spatial non-cooperative target image corresponds to position information of a spatial non-cooperative target and quaternion information of a spatial non-cooperative target;
[0037] The number of spatial non-cooperative target images, the number of spatial non-cooperative target position information, and the number of spatial non-cooperative target quaternion information are all N;
[0038] The specific process is:
[0039] The Coppeliasim simulation software was used to build a simulated satellite model, place a camera and set up a virtual space environment. The simulated satellite model was driven to flip along the motion trajectory within the camera's field of view through Python code programming to achieve a six-degree-of-freedom continuous transformation of the target. At the same time, the image of the space non-cooperative target, the position information of the space non-cooperative target and the quaternion information of the space non-cooperative target were saved.
[0040] Construct a training set (with labels) based on the spatial non-cooperative target image, the position information of the spatial non-cooperative target and the quaternion information of the spatial non-cooperative target;
[0041] A spatial non-cooperative target image corresponds to position information of a spatial non-cooperative target and quaternion information of a spatial non-cooperative target;
[0042] The number of spatial non-cooperative target images, the number of spatial non-cooperative target position information, and the number of spatial non-cooperative target quaternion information are all N;
[0043] The number of spatial non-cooperative target images, the amount of position information of spatial non-cooperative targets, and the amount of quaternion information of spatial non-cooperative targets are the same.
[0044] The other steps and parameters are the same as those in the first embodiment.
[0045] Specific implementation method three: This implementation method is different from specific implementation method one or two in that, in step two, the quaternion information (labeled) in the training set of step one is converted into Euler angles (labeled), and the Euler angles are encoded based on the soft classification idea to obtain the true value of the probability mass function of each Euler angle;
[0046] The specific process is:
[0047] 1) The quaternion in the training set of step 1 (labeled) conversion to Euler angles (labeled), i=1,2,.....N;
[0048] Where N represents the number of quaternion information in the training set;
[0049]
[0050] They represent the rotation angles of the spatial non-cooperative target around the X, Y, and Z axes (the simulation scene spatial coordinate system (set in the coppeliasim software));
[0051] 2) Encode the Euler angle based on the soft classification idea to obtain the probability mass function of each Euler angle; the specific process is:
[0052] 21) Design kernel function for Euler angles Encode and obtain the encoding result;
[0053] The kernel function is designed as shown in formula (1). Based on the soft classification idea, the Euler angle label is encoded and the geometric space is divided into M 3 Different sub-intervals, converting the direct regression of the pose quaternion into Euler angle classification;
[0054]
[0055] in, represents the encoding result; σ represents a constant used to adjust the output size; e p Indicates the set standard value of the Euler angle; M indicates the number of intervals into which the space is divided;
[0056] 22) Normalize the encoding result to obtain the true value of the probability mass function of each Euler angle.
[0057] The encoding result K(a i ,b) normalize, as shown in formula (2):
[0058]
[0059] Among them, E(a i ,b) represents the normalized encoding result, a i represents the Euler angle to be encoded; b represents the converted spatial interval variable.
[0060] The other steps and parameters are the same as those in the first or second embodiment.
[0061] Specific implementation method 4: This implementation method is different from one of the specific implementation methods 1 to 3 in that in step 3, a multi-scale dense structure pose estimation network is constructed to obtain a trained multi-scale dense structure pose estimation network; the specific process is:
[0062] Step 31: Construct a multi-scale dense structure pose estimation network;
[0063] Step 32: Construct the loss function L of the multi-scale dense structure pose estimation network estimation ;
[0064] Step 33: Loss function L based on the constructed multi-scale dense structure pose estimation network and scale dense structure pose estimation network estimation , obtain a trained multi-scale dense structure pose estimation network.
[0065] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.
[0066] Specific implementation method five: This implementation method is different from specific implementation methods one to four in that a multi-scale dense structure pose estimation network is constructed in step three-one; the specific process is:
[0067] The multi-scale dense structure pose estimation network includes: the first convolution layer, the first reluctant activation function layer, the multi-scale convolution module A, the multi-scale convolution module B, the multi-scale convolution module C, and the fully connected layer;
[0068] The multi-scale convolution module A includes: a second convolution layer with a convolution kernel size of 1×1, a second reluctant activation function layer, a third convolution layer with a convolution kernel size of 3×3, a third reluctant activation function layer, a fourth convolution layer with a convolution kernel size of 5×5, a fourth reluctant activation function layer, and a first BN layer;
[0069] The multi-scale convolution module B includes: a fifth convolution layer with a convolution kernel size of 1×1, a fifth reluctant activation function layer, a sixth convolution layer with a convolution kernel size of 3×3, a sixth reluctant activation function layer, a seventh convolution layer with a convolution kernel size of 5×5, a seventh reluctant activation function layer, and a second BN layer;
[0070] The multi-scale convolution module C includes: an eighth convolution layer with a convolution kernel size of 1×1, an eighth reluctant activation function layer, a ninth convolution layer with a convolution kernel size of 3×3, a ninth reluctant activation function layer, a tenth convolution layer with a convolution kernel size of 5×5, a tenth reluctant activation function layer, and a third BN layer;
[0071] The specific processing process of the multi-scale dense structure pose estimation network is as follows:
[0072] The spatial non-cooperative target image in the training set of step 1 is sequentially input into the first convolutional layer and the first reluctant activation function layer, and the first reluctant activation function layer outputs feature A;
[0073] Feature A is input into multi-scale convolution module A, and multi-scale convolution module A outputs feature B;
[0074] Features A and B are input into multi-scale convolution module B, and multi-scale convolution module B outputs feature C;
[0075] Features A, B and C are input into the multi-scale convolution module C, and the multi-scale convolution module C outputs feature D;
[0076] Feature D is input into the fully connected layer, and the fully connected layer outputs three branches, namely the plane position, depth and posture of the spatial non-cooperative target;
[0077] Among them, the plane position of the spatial non-cooperative target is the position of the target on the X and Y axes; the depth is the position of the target on the Z axis, and the posture is the predicted probability mass function.
[0078] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.
[0079] Specific implementation method 6: This implementation method is different from the specific implementation methods 1 to 5 in that the specific processing process of the multi-scale convolution module A is as follows:
[0080] Feature A is sequentially input into the second convolution layer with a convolution kernel size of 1×1 and the second Reluctant activation function layer, and the second Reluctant activation function layer outputs feature E;
[0081] Feature E is sequentially input into the third convolution layer with a convolution kernel size of 3×3 and the third Reluctant activation function layer, and the third Reluctant activation function layer outputs feature F;
[0082] Feature E is sequentially input into the fourth convolution layer with a convolution kernel size of 5×5 and the fourth reluctant activation function layer, and the fourth reluctant activation function layer outputs feature G;
[0083] Features E, F, and G are input into the first BN layer, and the output features of the first BN layer are used as the output feature B of the multi-scale convolution module A.
[0084] The other steps and parameters are the same as those in Specific Implementations 1 to 5.
[0085] Specific implementation method 7: This implementation method is different from any one of specific implementation methods 1 to 6 in that the specific processing process of the multi-scale convolution module B is as follows:
[0086] Feature A and feature B are sequentially input into the fifth convolution layer with a convolution kernel size of 1×1 and the fifth reluctant activation function layer, and the fifth reluctant activation function layer outputs feature H;
[0087] Feature H is sequentially input into the sixth convolution layer with a convolution kernel size of 3×3 and the sixth reluctant activation function layer, and the sixth reluctant activation function layer outputs feature I;
[0088] Feature H is sequentially input into the seventh convolution layer with a convolution kernel size of 5×5 and the seventh reluctant activation function layer, and the seventh reluctant activation function layer outputs feature J;
[0089] Features H, I, and J are input into the second BN layer, and the output features of the second BN layer are used as the output features C of the multi-scale convolution module B.
[0090] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.
[0091] Specific implementation eight: This implementation differs from any one of specific implementations one to seven in that the specific processing process of the multi-scale convolution module C is as follows:
[0092] Features A, B, and C are sequentially input into the eighth convolution layer with a convolution kernel size of 1×1 and the eighth reluctant activation function layer, and the eighth reluctant activation function layer outputs feature K;
[0093] Feature K is sequentially input into the ninth convolution layer with a convolution kernel size of 3×3 and the ninth reluctant activation function layer, and the ninth reluctant activation function layer outputs feature L;
[0094] Feature K is sequentially input into the tenth convolution layer with a convolution kernel size of 5×5 and the tenth reluctant activation function layer, and the tenth reluctant activation function layer outputs feature M;
[0095] Input features K, L, and M into the third BN layer, and the output features of the third BN layer are used as the output features D of the multi-scale convolution module C;
[0096] The design is based on a multi-scale convolution module, which uses convolution layers with kernel sizes of 1×1, 3×3, and 5×5 to convolve the input feature maps to obtain feature maps of different scales and the same size, and then combines these feature maps together in the channel dimension to complete the construction of the multi-scale convolution module.
[0097] Three multi-scale convolution modules with similar structures are connected in series, and a dense network structure is introduced. The output of each multi-scale convolution module is used as the input of each subsequent module to realize the construction of the multi-scale dense feature extraction network in the present invention, thereby obtaining multi-scale information of spatial non-cooperative target images and alleviating the problem of weak correlation between the front and rear convolution layers in the convolution network;
[0098] Based on the feature extraction network in the second step, a multi-scale dense structure pose estimation network is established. The network output contains three branches, which regress and output the plane position, depth and posture of the target respectively. The plane position includes the position of the target on the X and Y axes in the dataset, the depth refers to the position on the Z axis, and the output posture refers to the probability mass function after the quaternion label encoding of the dataset in the first step, according to the different division intervals in the second step.
[0099] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.
[0100] Specific implementation method 9: This implementation method is different from the specific implementation methods 1 to 8 in that the loss function L of the multi-scale dense structure pose estimation network is constructed in step 32. estimation ; The specific process is:
[0101] The loss function L of the multi-scale dense structure pose estimation network estimation As shown below:
[0102] L estimation =L xy +L deep +L attitude (3)
[0103] Where, L attitude represents the difference between the probability mass function output by the multi-scale dense structure pose estimation network in step 3 and the true value of the probability mass function obtained in step 2; L xyrepresents the plane position loss function; L deep Loss function representing depth position;
[0104] L attitude Used to measure the quality of network pose estimation;
[0105] Step 3: The difference between the probability mass function output by the multi-scale dense structure pose estimation network and the true value of the probability mass function obtained in step 2 attitude It is expressed as follows:
[0106]
[0107] In the formula, represents the true value of the probability mass function of the i-th Euler angle obtained in step 2; represents the predicted probability mass function of the output of the multi-scale dense structure pose estimation network when the i-th spatial non-cooperative target image in the training set of step 1 is input;
[0108] Considering the influence of depth information on position estimation, it is not reasonable to use absolute error to measure the position error and size of non-cooperative targets. Therefore, the present invention designs a loss function L that is linearly related to the target depth information. xy ;
[0109] Plane position loss function L xy The expression is:
[0110]
[0111] in, It indicates the position of the predicted i-th spatial non-cooperative target image in the X and Y directions output by the multi-scale dense structure pose estimation network inputting the i-th spatial non-cooperative target image in the training set of step 1; Represents the true value of the position of the i-th spatial non-cooperative target image in the training set of step 1 in the X and Y directions; represents the true value of the position of the i-th spatial non-cooperative target image in the Z direction in the training set of step 1; ||·|| represents the norm;
[0112] The output branch of the pose estimation network of the present invention includes the regression of the Z-axis information, and the loss function is designed by using the 2-norm to train the model's regression ability for depth information, as shown in formula (6);
[0113] The loss function L of the depth position deep The expression is:
[0114]
[0115] in, It indicates the position of the i-th spatial non-cooperative target image in the Z direction predicted by the multi-scale dense structure pose estimation network output by inputting the i-th spatial non-cooperative target image in the training set of step 1; represents the true value of the position of the i-th spatial non-cooperative target image in the training set of step 1 in the Z direction; ||·||2 represents the binary norm.
[0116] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.
[0117] Specific implementation ten: This implementation differs from any one of specific implementations one to nine in that in step four, the non-cooperative target image in the space to be measured is input into a trained multi-scale dense structure pose estimation network, and the trained multi-scale dense structure pose estimation network outputs a predicted probability mass function;
[0118] Decoding the predicted probability mass function to obtain the decoded Euler angle;
[0119] The decoded Euler angle Follow step 2 to convert back to quaternion format Will Combined with the position of the spatial non-cooperative target on the X, Y, and Z axes output by the trained multi-scale dense structure pose estimation network, the six-degree-of-freedom pose information of the spatial non-cooperative target is obtained;
[0120] The specific process is:
[0121] The image of the non-cooperative spatial target to be tested is input into the trained multi-scale dense structure pose estimation network, and the trained multi-scale dense structure pose estimation network outputs three branches, which are the plane position, depth and pose of the non-cooperative spatial target;
[0122] Among them, the plane position of the spatial non-cooperative target is the position of the target on the X and Y axes; the depth is the position of the target on the Z axis, and the posture is the predicted probability mass function;
[0123] The predicted probability mass function is decoded to obtain the decoded Euler angle. The decoding formula is as follows:
[0124]
[0125] In formula (7), M represents the number of intervals of space division, k represents the kth interval of division, ω k ={ω1,ω2,...,ω M3} represents the predicted probability mass function of the output of the trained multi-scale dense structure pose estimation network; e represents the Euler angle of the fitting process; Represents the decoded Euler angle;
[0126] The decoded Euler angle Follow step 2 to convert back to quaternion format Will Combined with the position of the spatial non-cooperative target on the X, Y, and Z axes output by the trained multi-scale dense structure pose estimation network, the six-degree-of-freedom pose information of the spatial non-cooperative target is obtained.
[0127] The other steps and parameters are the same as those in Specific Embodiments 1 to 9.
[0128] Save the pose estimation model trained in the third and fourth steps, input the test set image in the first step into the saved pose estimation model, and decode the probability mass function output by the model to obtain the non-cooperative target pose information;
[0129] After the training is completed, the model performance is evaluated, that is, the pose estimation accuracy. The evaluation index includes two parts, which represent the evaluation of the pose estimation accuracy and the evaluation of the position estimation accuracy, as shown in equations (8) and (9).
[0130]
[0131]
[0132] where q gt and P gt Represent the true value of the spatial non-cooperative target posture and position, and Represents the posture and position prediction values output by the network.
[0133] Example:
[0134] In a complex space environment, a spatial non-cooperative target pose estimation method based on a multi-scale dense structure neural network is proposed. The quaternion label is encoded by soft classification, and the traditional end-to-end pose estimation network regression problem is transformed into a classification problem, achieving high-precision pose estimation. The overall pose estimation process is as follows: Figure 1 shown.
[0135] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings.
[0136] Step 1: Create a simulated satellite model in Coppeliasim. This step involves designing the shape, size, and possible dynamic components of the satellite (such as solar panels); adding a virtual camera in the Coppeliasim environment to simulate the functions of real-world camera equipment. The position and angle of the camera need to be set accurately to ensure that the satellite model is within its field of view. The images collected by the virtual camera are as follows: Figure 4As shown; a realistic space environment is constructed to simulate the situation in space more realistically; by writing Python code, the simulated satellite model can flip along a predetermined trajectory in the virtual environment. This part of the code will control the continuous transformation of the six degrees of freedom (front and back, left and right, up and down movement, and rotation around three axes) of the satellite model; during the simulation process, Python code is used to automatically capture the image of the satellite model and record its position information and quaternion information in the virtual space; the captured images and their related information are organized into a structured data set, and then divided according to the ratio of 8:2, that is, 80% of the data is used as a training set and 20% of the data is used as a test set. The training set is used to train the machine learning model, and the test set is used to evaluate the performance of the model.
[0137] Step 2: Convert the quaternion labels of the data set created in the first step into the form of Euler angles, design a kernel function, encode the Euler angle labels based on the soft classification idea, and divide the geometric space into M 3 Different subintervals are encoded using the label and the space to be transformed using the kernel function, the regression problem is converted into an Euler angle classification problem, and then the encoding results are normalized.
[0138] Step 3: First, design a module based on multi-scale convolution kernels. Use convolution kernels of different sizes, namely 1×1, 3×3 and 5×5, to process the input feature maps, thereby capturing local feature information of various sizes. By performing convolution operations, three sets of feature maps of different scales but the same size are generated; these feature maps of different scales are merged along the channel dimension to obtain rich multi-scale feature representations. The multi-scale convolution module is as follows: Figure 2 shown.
[0139] Arrange three multi-scale convolution modules with similar structures in sequence, and then design a dense connection architecture, that is, the output of the previous multi-scale convolution module is not only the input of the next module, but also a part of the input of all subsequent modules, thus forming a multi-scale dense feature extraction network, such as Figure 3 shown.
[0140] Based on the multi-scale dense feature extraction network, the present invention establishes a pose estimation network to simultaneously predict the plane position (X, Y axis coordinates), depth (Z axis position) and pose of a non-cooperative target in space, and designs three regression branches for accurate estimation. The plane position and depth directly reflect the position of the target in three-dimensional space, and the pose branch regresses the probability mass function to describe the pose information of the target.
[0141] Step 4: Train the multi-scale dense structure pose estimation network in the third step, and design the training optimization process and loss function to ensure that the network can accurately estimate the plane position, depth and pose of the target. For the pose estimation network of non-cooperative targets in space, the design of the loss function needs to consider the estimation accuracy of position, depth and pose at the same time. Therefore, the present invention proposes a composite loss function as shown in formula (3), which consists of three parts: plane position loss, depth loss and pose loss.
[0142] The present invention introduces random rotation, translation and scaling data enhancement technology to enhance the generalization ability and robustness of the network; by adopting a cosine decay strategy for the learning rate, the learning rate is gradually reduced to refine the network parameters, and the Adam optimizer is used to train the model.
[0143] Step 5: After designing and training the multi-scale dense structure pose estimation network, it is necessary to decode the probability mass function output by the network and convert it back into the form of quaternion or Euler angle, that is, to recover the pose information from the discrete probability. The decoding method is shown in formula (7).
[0144]
[0145] In formula (7), M represents the number of intervals divided into spaces, k represents the kth interval divided, represents the predicted probability mass function of the output of the trained multi-scale dense structure pose estimation network; e represents the Euler angle of the fitting process; Represents the decoded Euler angle;
[0146] Step 6: The present invention uses formula (8) and formula (9) to judge the model, which represent the evaluation indicators of attitude estimation accuracy and position estimation accuracy respectively.
[0147] The evaluation of posture is shown in formula (8), which is achieved by calculating the angular difference between the predicted posture and the true posture.
[0148]
[0149] where q gt represents the true value of the spatial non-cooperative target posture, Represents the posture prediction value output by the network. The returned value is radians, which needs to be converted to degrees.
[0150] The evaluation of position estimation is usually performed by calculating the Euclidean distance between the predicted position and the true position, as shown in equation (9).
[0151]
[0152] Where P gtrepresents the true value of the spatial non-cooperative target position, Represents the location prediction value output by the network.
[0153] The present invention has completed the overall programming of the program, and only the images contained in the data set in the first step are needed to train the model. A complete processing process is given below to further illustrate the present invention. The simulation conditions are: Windows 10 operating system, Pycharm software, Coppeliasim software.
[0154] The image data collected by the virtual camera arranged in the Coppeliasim simulation environment are preprocessed, multi-scale dense structure feature extraction, 3-DOF position and attitude probability mass function regression, and decoding network output probability mass function to finally obtain the 6-DOF position and attitude information of the spatial non-cooperative target. The program includes Multi_conv_model.py, train.py, test.py, Generate_bins.py, and utils.py files.
[0155] 1. Simulation environment construction and data set establishment
[0156] SolidWorks was used to model the non-cooperative target model in space, which was imported into the coppeliasim simulation software to write the motion trajectory using Python. The image was saved and the position and attitude quaternion true value of the model during movement was recorded to form a json file and establish a data set.
[0157] 2. Soft encoding of dataset pose labels
[0158] (1) Divide space
[0159] The space is divided into 4×4×4, a total of 64 parts, and 64 different intervals are generated. The generate_bins() function is used to generate the intervals in a loop. The quaternion in the label is converted into the Euler angle form in the order of xyz using the quat2euler(quaternion) function. quat2euler(quaternion) contains r=R.from_quat(quaternion), where quaternion represents the quaternion to be converted.
[0160] (2) Design kernel function and encode labels
[0161] First, the kernel function in formula (1) is written as a program by designing the K_function(x,y,delta) function, and then each label and the divided space are encoded to convert the Euler angle label into a probability mass function.
[0162] 3. Pose Estimation Network Model Design
[0163] (1) Multi-scale convolution module
[0164] Design the Multi_conv_module (nn.Module) class, and initialize it including input_channel, output_channel, stride, such as Figure 2 As shown. First, create three convolutional layers conv1, conv2, and conv3, whose convolution kernels are 1, 3, and 5 respectively. In order to ensure the same size of the output feature layer, the padding of these three convolutional layers is set to 0, 1, and 2 respectively; input the feature map to the three branches respectively, and then use the torch.cat() function to combine the outputs of the three convolutional layers in the channel dimension, and then add nn.ReLU() and nn.BatchNorm2d() to complete the activation function and normalization.
[0165] (2) Multi-scale dense structure pose estimation network
[0166] First, we design the convolution layer pre_conv to expand the monocular image from a single channel to 64 channels, and then define three multi-scale convolution modules. In the definition of the forward(self, input) function, as follows Figure 3 The dense structure shown in the figure is designed, and the output of each multi-scale convolution module is used as the input of the next convolution module, thereby completing the feature extraction of spatial non-cooperative target monocular images.
[0167] Then, three fully connected layers are designed, namely self.loc_fc_xy(), self.loc_fc_z(), and self.ori_fc(). Among them, self.loc_fc_xy() and self.loc_fc_z() are composed of three nn.Linear() and two nn.Dropout(), and finally output a 1-dimensional vector; self.ori_fc() outputs a 64-dimensional vector, which represents the probability mass function predicted by the network.
[0168] 4. Network model training
[0169] Use build_train_val_dataloader(speed_root,data_transforms,batch_size) in utils.py to build the dataset, where speed_root represents the dataset directory, data_transforms represents the image preprocessing module, and batch_size represents the batch of each training.
[0170] Write code based on the loss function in equations (3)-(6), use the network output and the true value of the label to calculate the loss function and perform back propagation to complete the iterative optimization process of the model.
[0171] V. Probability mass function of decoding network output
[0172] After network prediction, it is necessary to decode the probability mass function of the network output. Minimize() is introduced from scipy.optimize, and minimize(func_of_bins, init_bins, args = num_bins, method = 'SLSQP', options = {'ftol': 1e-20, 'disp': False}) is designed to complete the decoding, where func_of_bins represents the function to be solved, init_bins represents the initialization value, args represents the parameter, method represents the solution method, and options is used to control the number of iterations, which is designed in the form of a dictionary.
[0173] 6. Evaluate the experimental results
[0174] According to equations (8) and (9), the evaluation index codes are written to evaluate the network's position and attitude estimation accuracy respectively. The position and attitude estimation accuracy is shown in Table 1. For a more intuitive representation, the quaternion error is converted into Euler angle representation. Figure 5 This is the visualization result of the pose estimation for a non-cooperative target in space according to the present invention. The polar coordinate diagram on the right represents the angles of rotation around the XYZ axes, where blue represents the true value and red represents the predicted value.
[0175] Table 1: Pose estimation results
[0176]
[0177] It can be seen from the evaluation indicators of the pose estimation results that the pose estimation method based on the multi-scale dense structure neural network proposed in the present invention can achieve high-precision pose estimation of six degrees of freedom for space non-cooperative targets in a space environment, providing technical support for subsequent space on-orbit missions.
[0178] The present invention may also have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.
Claims
1. A spatial non-cooperative target pose estimation method based on a multi-scale dense structure neural network, characterized by: The specific process of the method is: Step 1: Acquire the image of the spatial non-cooperative target, the position information of the spatial non-cooperative target, and the quaternion information of the spatial non-cooperative target; A training set is constructed based on the spatial non-cooperative target image, the position information of the spatial non-cooperative target, and the quaternion information of the spatial non-cooperative target; A spatial non-cooperative target image corresponds to position information of a spatial non-cooperative target and quaternion information of a spatial non-cooperative target; The number of spatial non-cooperative target images, the number of spatial non-cooperative target position information, and the number of spatial non-cooperative target quaternion information are all N; Step 2: Convert the quaternion information in the training set of step 1 into Euler angles, encode the Euler angles based on the soft classification idea, and obtain the true value of the probability mass function of each Euler angle; Step 3: Construct a multi-scale dense structure pose estimation network to obtain a trained multi-scale dense structure pose estimation network; Step 4: Input the non-cooperative target image in the tested space into the trained multi-scale dense structure pose estimation network, and the trained multi-scale dense structure pose estimation network outputs the predicted probability mass function; Decoding the predicted probability mass function to obtain the decoded Euler angle; The decoded Euler angle Follow step 2 to convert back to quaternion format Will Combined with the position of the spatial non-cooperative target on the X, Y, and Z axes output by the trained multi-scale dense structure pose estimation network, the six-degree-of-freedom pose information of the spatial non-cooperative target is obtained.
2. The spatial non-cooperative target pose estimation method based on a multi-scale dense structure neural network according to claim 1, characterized in that: In the step 1, the spatial non-cooperative target image, the position information of the spatial non-cooperative target and the quaternion information of the spatial non-cooperative target are obtained; A training set is constructed based on the spatial non-cooperative target image, the position information of the spatial non-cooperative target, and the quaternion information of the spatial non-cooperative target; A spatial non-cooperative target image corresponds to position information of a spatial non-cooperative target and quaternion information of a spatial non-cooperative target; The number of spatial non-cooperative target images, the number of spatial non-cooperative target position information, and the number of spatial non-cooperative target quaternion information are all N; The specific process is: The Coppeliasim simulation software was used to build a simulated satellite model, place a camera and set up a virtual space environment. The simulated satellite model was driven to flip along the motion trajectory within the camera's field of view through Python code programming to achieve the target six-degree-of-freedom continuous transformation. At the same time, the image of the spatial non-cooperative target, the position information of the spatial non-cooperative target and the quaternion information of the spatial non-cooperative target are saved; A training set is constructed based on the spatial non-cooperative target image, the position information of the spatial non-cooperative target, and the quaternion information of the spatial non-cooperative target; A spatial non-cooperative target image corresponds to position information of a spatial non-cooperative target and quaternion information of a spatial non-cooperative target; The number of spatial non-cooperative target images, the number of position information of spatial non-cooperative targets, and the number of quaternion information of spatial non-cooperative targets are all N.
3. The spatial non-cooperative target pose estimation method based on a multi-scale dense structure neural network according to claim 2, characterized in that: In the step 2, the quaternion information in the training set of step 1 is converted into Euler angles, and the Euler angles are encoded based on the soft classification idea to obtain the true value of the probability mass function of each Euler angle; The specific process is: 1) The quaternion in the training set of step 1 Convert to Euler angles i=1,2,.....N; Where N represents the number of quaternion information in the training set; 2) Encode the Euler angle based on the soft classification idea to obtain the probability mass function of each Euler angle; the specific process is: 21) Design kernel function for Euler angles Encode and obtain the encoding result; in, represents the encoding result; σ represents a constant; e p Indicates the set standard value of the Euler angle; M indicates the number of intervals into which the space is divided; 22) Normalize the encoding result to obtain the true value of the probability mass function of each Euler angle.
4. The spatial non-cooperative target pose estimation method based on a multi-scale dense structure neural network according to claim 3, characterized in that: In the step 3, a multi-scale dense structure pose estimation network is constructed to obtain a trained multi-scale dense structure pose estimation network; the specific process is: Step 31: Construct a multi-scale dense structure pose estimation network; Step 32: Construct the loss function L of the multi-scale dense structure pose estimation network estimation ; Step 33: Loss function L based on the constructed multi-scale dense structure pose estimation network and scale dense structure pose estimation network estimation , obtain a trained multi-scale dense structure pose estimation network.
5. The method for spatial non-cooperative target pose estimation based on a multi-scale dense structure neural network according to claim 4, characterized in that: In the step 31, a multi-scale dense structure pose estimation network is constructed; The specific process is: The multi-scale dense structure pose estimation network includes: the first convolution layer, the first reluctant activation function layer, the multi-scale convolution module A, the multi-scale convolution module B, the multi-scale convolution module C, and the fully connected layer; The multi-scale convolution module A includes: a second convolution layer with a convolution kernel size of 1×1, a second reluctant activation function layer, a third convolution layer with a convolution kernel size of 3×3, a third reluctant activation function layer, a fourth convolution layer with a convolution kernel size of 5×5, a fourth reluctant activation function layer, and a first BN layer; The multi-scale convolution module B includes: a fifth convolution layer with a convolution kernel size of 1×1, a fifth reluctant activation function layer, a sixth convolution layer with a convolution kernel size of 3×3, a sixth reluctant activation function layer, a seventh convolution layer with a convolution kernel size of 5×5, a seventh reluctant activation function layer, and a second BN layer; The multi-scale convolution module C includes: an eighth convolution layer with a convolution kernel size of 1×1, an eighth reluctant activation function layer, a ninth convolution layer with a convolution kernel size of 3×3, a ninth reluctant activation function layer, a tenth convolution layer with a convolution kernel size of 5×5, a tenth reluctant activation function layer, and a third BN layer; The specific processing process of the multi-scale dense structure pose estimation network is as follows: The spatial non-cooperative target image in the training set of step 1 is sequentially input into the first convolutional layer and the first reluctant activation function layer, and the first reluctant activation function layer outputs feature A; Feature A is input into multi-scale convolution module A, and multi-scale convolution module A outputs feature B; Features A and B are input into the multi-scale convolution module B, and the multi-scale convolution module B outputs feature C; Features A, B and C are input into the multi-scale convolution module C, and the multi-scale convolution module C outputs feature D; Feature D is input into the fully connected layer, and the fully connected layer outputs three branches, namely the plane position, depth and posture of the spatial non-cooperative target; Among them, the plane position of the spatial non-cooperative target is the position of the target on the X and Y axes; the depth is the position of the target on the Z axis, and the posture is the predicted probability mass function.
6. The method for spatial non-cooperative target pose estimation based on a multi-scale dense structure neural network according to claim 5, characterized in that: The specific processing process of the multi-scale convolution module A is as follows: Feature A is sequentially input into the second convolution layer with a convolution kernel size of 1×1 and the second Reluctant activation function layer, and the second Reluctant activation function layer outputs feature E; Feature E is sequentially input into the third convolution layer with a convolution kernel size of 3×3 and the third Reluctant activation function layer, and the third Reluctant activation function layer outputs feature F; Feature E is sequentially input into the fourth convolution layer with a convolution kernel size of 5×5 and the fourth reluctant activation function layer, and the fourth reluctant activation function layer outputs feature G; Features E, F, and G are input into the first BN layer, and the output features of the first BN layer are used as the output feature B of the multi-scale convolution module A.
7. The method for spatial non-cooperative target pose estimation based on a multi-scale dense structure neural network according to claim 6, characterized in that: The specific processing process of the multi-scale convolution module B is: Feature A and feature B are sequentially input into the fifth convolution layer with a convolution kernel size of 1×1 and the fifth reluctant activation function layer, and the fifth reluctant activation function layer outputs feature H; Feature H is sequentially input into the sixth convolution layer with a convolution kernel size of 3×3 and the sixth reluctant activation function layer, and the sixth reluctant activation function layer outputs feature I; Feature H is sequentially input into the seventh convolution layer with a convolution kernel size of 5×5 and the seventh reluctant activation function layer, and the seventh reluctant activation function layer outputs feature J; Features H, I, and J are input into the second BN layer, and the output features of the second BN layer are used as the output features C of the multi-scale convolution module B.
8. The method for spatial non-cooperative target pose estimation based on a multi-scale dense structure neural network according to claim 7, characterized in that: The specific processing process of the multi-scale convolution module C is: Features A, B, and C are sequentially input into the eighth convolution layer with a convolution kernel size of 1×1 and the eighth reluctant activation function layer, and the eighth reluctant activation function layer outputs feature K; Feature K is sequentially input into the ninth convolution layer with a convolution kernel size of 3×3 and the ninth reluctant activation function layer, and the ninth reluctant activation function layer outputs feature L; Feature K is sequentially input into the tenth convolution layer with a convolution kernel size of 5×5 and the tenth reluctant activation function layer, and the tenth reluctant activation function layer outputs feature M; The features K, L, and M are input into the third BN layer, and the output features of the third BN layer are used as the output features D of the multi-scale convolution module C.
9. The spatial non-cooperative target pose estimation method based on a multi-scale dense structure neural network according to claim 8, characterized in that: The loss function L of the multi-scale dense structure pose estimation network constructed in step 32 is estimation ; The specific process is: The loss function L of the multi-scale dense structure pose estimation network estimation As shown below: L estimation =L xy +L deep +L attitude (3) Where, L attitude represents the difference between the probability mass function output by the multi-scale dense structure pose estimation network in step 3 and the true value of the probability mass function obtained in step 2; L xy represents the plane position loss function; L deep Loss function representing depth position; Step 3: The difference between the probability mass function output by the multi-scale dense structure pose estimation network and the true value of the probability mass function obtained in step 2 attitude It is expressed as follows: In the formula, represents the true value of the probability mass function of the i-th Euler angle obtained in step 2; represents the predicted probability mass function of the output of the multi-scale dense structure pose estimation network when the i-th spatial non-cooperative target image in the training set of step 1 is input; Plane position loss function L xy The expression is: in, It indicates the position of the predicted i-th spatial non-cooperative target image in the X and Y directions output by the multi-scale dense structure pose estimation network inputting the i-th spatial non-cooperative target image in the training set of step 1; Represents the true value of the position of the i-th spatial non-cooperative target image in the training set of step 1 in the X and Y directions; represents the true value of the position of the i-th spatial non-cooperative target image in the Z direction in the training set of step 1; ||·|| represents the norm; The loss function L of the depth position deep The expression is: in, It indicates the position of the i-th spatial non-cooperative target image in the Z direction predicted by the multi-scale dense structure pose estimation network output by inputting the i-th spatial non-cooperative target image in the training set of step 1; represents the true value of the position of the i-th spatial non-cooperative target image in the training set of step 1 in the Z direction; ||·||2 represents the binary norm.
10. The method for spatial non-cooperative target pose estimation based on a multi-scale dense structure neural network according to claim 9, characterized in that: In the step 4, the non-cooperative target image in the space to be measured is input into a trained multi-scale dense structure pose estimation network, and the trained multi-scale dense structure pose estimation network outputs a predicted probability mass function; Decoding the predicted probability mass function to obtain the decoded Euler angle; The decoded Euler angle Follow step 2 to convert back to quaternion format Will Combined with the position of the spatial non-cooperative target on the X, Y, and Z axes output by the trained multi-scale dense structure pose estimation network, the six-degree-of-freedom pose information of the spatial non-cooperative target is obtained; The specific process is: The image of the non-cooperative spatial target to be tested is input into the trained multi-scale dense structure pose estimation network, and the trained multi-scale dense structure pose estimation network outputs three branches, which are the plane position, depth and pose of the non-cooperative spatial target; Among them, the plane position of the spatial non-cooperative target is the position of the target on the X and Y axes; the depth is the position of the target on the Z axis, and the posture is the predicted probability mass function; The predicted probability mass function is decoded to obtain the decoded Euler angle. The decoding formula is as follows: In formula (7), M represents the number of intervals divided into space, k represents the kth interval divided, represents the predicted probability mass function of the output of the trained multi-scale dense structure pose estimation network; e represents the Euler angle of the fitting process; Represents the decoded Euler angle; The decoded Euler angle Follow step 2 to convert back to quaternion format Will Combined with the position of the spatial non-cooperative target on the X, Y, and Z axes output by the trained multi-scale dense structure pose estimation network, the six-degree-of-freedom pose information of the spatial non-cooperative target is obtained.
Citation Information
Patent Citations
Spatial non-cooperative target relative pose estimation method based on deep learning
CN111862201A
Spatial non-cooperative target pose estimation method based on deep learning
CN112651437A
Cited By
Non-cooperative satellite attitude measurement method based on sequence images
CN122130102A
