Space target relative pose estimation deep reinforcement learning method based on linear features

Through deep reinforcement learning technology, design state and action space, and using DDPG network training, the problem of difficult linear feature matching relationship in the spatial environment is solved, accurate target relative pose estimation is achieved, and on-orbit service and space debris removal is improved The accuracy of the task.

CN118982649APending Publication Date: 2024-11-19SUZHOU SANYUAN AEROSPACE TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410782401.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-17
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

In a spatial environment, due to factors such as lighting and noise, it is difficult to accurately estimate target relative pose parameters, especially when the linear feature matching relationship is unknown, the prior art is difficult to effectively solve.

Method used

Using deep reinforcement learning technology, through designing state space, observation space, action space and reward functions, DDPG network training obtains the correspondence relationship between straight lines and target relative poses, and establishes the agent to adjust the position and posture to minimize the model. The distance between a straight line and a straight line on the image plane.

Benefits of technology

Accurate pose recognition within different initial value ranges is achieved, the accuracy of relative pose estimation of spatial targets is improved, the difficulty of automatically obtaining the linear matching relationship between the image plane and the target model is overcome, and satisfactory pose information is obtained.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982649A_ABST
    Figure CN118982649A_ABST
Patent Text Reader

Abstract

A space target relative pose estimation deep reinforcement learning method based on linear features comprises the steps of S1) obtaining a target model straight line, S2) carrying out straight line extraction on a target image, S3) establishing an intelligent body (virtual target), and S4) obtaining an intelligent body (virtual target). S4) establishing a distance matrix by using the current position and attitude of the virtual target; s5) calculating a current state reward value; s6) if the reward value in the step S5) does not reach the set threshold value, training a critic network by using a new reward value and a Q value, and updating an actor network weight; and S7) obtaining the next action through the actor network, updating the position and the posture, and returning to the step 4). According to the method, a virtual target is established to serve as an intelligent agent, projection geometric transformation is integrated into the Markov process, and the matching relation between straight lines is obtained through reinforcement learning to estimate the relative pose of the target. Simulation verification is carried out on the method, and the pose recognition precision reaches a satisfactory level.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of on-orbit services in the aerospace field, and relates to target posture recognition technology in close-range observation tasks, and in particular to a deep reinforcement learning method for relative posture estimation of space targets based on straight line features. Background Art

[0002] Accurately estimating the relative pose parameters of the target is the key to successfully completing tasks such as on-orbit servicing and space debris removal. Visual sensors are widely used in target recognition and pose estimation due to their low cost, low power consumption, and no need for cooperative information. The traditional method of vision-based target pose measurement is through template matching, feature matching, and tracking. The typical PnP algorithm can calculate the position and pose of the target relative to the camera coordinate system with high accuracy by obtaining the position of the target feature points in the imaging plane and the corresponding position of the body coordinate system.

[0003] Due to the influence of factors such as lighting and noise in the space environment, imaging feature points are easily lost. For the surface of artificial targets in space, straight line features are relatively abundant, and straight line detection and positioning are more reliable, with richer attributes and more stable matching. Therefore, the PnL method was proposed: when the correspondence between the model straight line and the image projection straight line is known, the relative position and posture of the target can be solved by least squares and maximum likelihood estimation. When the matching relationship between straight lines is unknown, this problem is difficult to solve. In the field of computer vision, this problem is called the instant posture and matching solution problem. A common algorithm for this problem is the iterative solution method, which reduces posture solution and matching to an optimization problem, and alternately solves the current optimal posture and the optimal match. However, due to the existence of many local extreme values, when the gap between the initial posture selected by the iterative algorithm and the true posture is large, the algorithm cannot give a correct solution;

[0004] Therefore, the existing technology needs to be improved. Summary of the invention

[0005] Purpose of the invention: In order to overcome the above shortcomings, the purpose of the present invention is to provide a deep reinforcement learning method for estimating the relative pose of spatial targets based on straight line features. In view of the difficulty in automatically obtaining the matching relationship between the image plane and the straight line of the target model, deep reinforcement learning technology is adopted to innovatively design the state space, observation space, action space and reward function of the system, and the correspondence between the straight lines and the relative pose of the target are obtained through DPG network training.

[0006] Technical solution: In order to achieve the above-mentioned purpose, the present invention provides a deep reinforcement learning method for relative pose estimation of space targets based on straight line features, comprising the following steps:

[0007] S1): Obtain the target model straight line, that is, select the key straight line in the target model as the model straight line, and the model straight line is represented by the position table of the two end points of the straight line in the body coordinate system;

[0008] S2): Acquire the target image to extract the straight line, that is, capture the target image by a camera, extract the straight line from the target area of ​​the image, and obtain the image straight line;

[0009] S3): Establish an intelligent agent, set the relative position and initial posture of the virtual target, as well as the initial action, and set the initial value of the action;

[0010] S4): Use the current position and posture of the virtual target to establish a distance matrix and observe its state; S5): Calculate the current state reward value. If the reward value reaches the set threshold, output the target relative position and straight line matching relationship; if the preset number of learning times is reached, the process ends;

[0011] S6): If the reward value in S5) does not reach the set threshold, the critic network is trained with the new reward value and Q value, and the actor network weight is updated;

[0012] S7): Get the next action through the actor network, update the position and posture, and return to step 4).

[0013] The deep reinforcement learning method for estimating relative pose of a space target based on straight line features described in the present invention, in said S3, an intelligent agent is established, the relative pose of the target and the initial value of the pose, as well as the initial action, are set, and the initial value of the action is set, as follows:

[0014] S0={rot0[3],tran0[3]} (1).

[0015] The deep reinforcement learning method for relative position and posture estimation of space targets based on straight line features described in the present invention, the specific process of establishing a distance matrix using the current position and posture of the virtual target in S4) is as follows:

[0016] S41): First, solve the model straight line projection corresponding to the virtual target in the current position:

[0017] The model line is represented by two endpoints. Let P be the endpoint of the model line, and the target's posture and position relative to the camera coordinate system are R and T respectively. Then the position of P in the camera coordinate system can be expressed as:

[0018]

[0019] The projection point of P on the image plane is p = (p u , p v ) can be expressed as:

[0020]

[0021] Among them, f x , f y , c x , c x are the focal length and optical center position of the camera in the x and y directions respectively;

[0022] S42): Solve the distance between the model line projection and the plane line:

[0023] Known target model 3D straight line {L i , i = 1, 2, ..., m}, image plane straight line {l j , j = 1, 2, ..., n}, straight line L i It is represented by two endpoints The projections of the two endpoints on the image plane are p1 and i , p2 i ,

[0024]

[0025] Define the distance d between the model projection line i and the detection line j in the image plane ij is the model line L i The distances between the projection points p1 and p2 of the two endpoints on the image plane and the detection line l are denoted by s1 and s2, respectively. , express:

[0026] d ij =0.5*(s1+s2) (5);

[0027] S43): Establish a distance matrix:

[0028] The distance matrix between the model lines and the plane lines is an mxn dimensional matrix, where m is the number of model lines and n is the number of plane lines, m>4, n>4;

[0029] O=D={d ij ,i=1,2,...,m,j=1,2,...,n} (6)

[0030] Among them, d ij Represents the distance between model line i and plane line j.

[0031] In the method for deep reinforcement learning of relative pose estimation of space targets based on straight line features described in the present invention, in the process of solving the distance between the model straight line projection and the plane straight line in S42), due to the discontinuity and elongation of the straight line extracted from the image, the image straight line l j It can be expressed as the equation of a plane line:

[0032] a j x+b j y+c j =0(7)

[0033] A point p(x c ,y c ) to the plane line l j The distance s can be expressed as:

[0034] s=|a j x c +b j y c +c j | (8).

[0035] The deep reinforcement learning method for relative pose estimation of space targets based on straight line features described in the present invention, wherein the current state reward value is calculated in S5), and the reward value for each step is the sum of each row in the observation matrix after executing the step, except for the minimum value selected, that is,

[0036]

[0037] match(k) represents the column number of the minimum value of the k-th row of matrix O;

[0038] At this time, if the reward value exceeds the set threshold, the target position and posture estimation values ​​are output;

[0039] If the exit condition of the reinforcement learning process is met, the process ends.

[0040] The deep reinforcement learning method for relative position and posture estimation of space targets based on straight line features described in the present invention, the S7): obtaining the next action through the actor network, updating the position and posture, is as follows:

[0041] S'=S+action(10).

[0042] It can be seen from the above technical solution that the present invention has the following beneficial effects:

[0043] 1. The deep reinforcement learning method for estimating the pose of a space target based on straight line features described in the present invention establishes a virtual space target as an intelligent agent, designs a reward function related to the distance between the model projection line and the image line, obtains the maximum reward by moving the position of the intelligent agent and rotating the posture, and finally obtains the correspondence between the model line and the plane line, and obtains the relative pose result that meets the accuracy requirements. By using this method to simulate and verify space targets with different initial value ranges, the pose recognition accuracy reaches a satisfactory level.

[0044] 2. In order to solve the problem that the matching relationship between the image plane and the straight line of the target model is difficult to obtain automatically, the present invention sets a virtual current as an intelligent agent, adopts deep reinforcement learning technology, and innovatively designs the state space, observation space, action space and reward function of the system. The virtual target is allowed to continuously approach the real target through position and posture changes (agent action), and the correspondence between the straight lines and the relative posture of the target are obtained through DPG network training, thereby obtaining posture information consistent with the target in the image.

[0045] 3. The present invention integrates the projective geometric transformation into the deep reinforcement learning Markov process by establishing a mathematical representation of the matching between the model straight line and the image straight line. Through learning, the matching relationship between the straight lines is obtained and the relative position of the target is accurately estimated.

[0046] 4. The deep reinforcement learning method for spatial target pose estimation based on straight line features described in the present invention can minimize the distance between the projection of the model straight line on the image plane and the corresponding plane straight line after the reinforcement learning process is completed, thereby obtaining the correspondence between the straight lines and the optimal pose parameter estimation value. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flow chart of a deep reinforcement learning method for estimating spatial target pose based on straight line features involved in the present invention;

[0048] Figure 2 It is the geometric relationship of straight line projection;

[0049] Figure 3 Define the distance between the model straight line and the plane straight line;

[0050] Figure 4 Simulation model and its coordinate definition;

[0051] Figure 5 DDPG training reward function change trend chart;

[0052] Figure 6 A snapshot of the agent's progress toward the true target during training. DETAILED DESCRIPTION

[0053] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments.

[0054] Example

[0055] like Figures 1 to 6 A deep reinforcement learning method for relative pose estimation of a space target based on straight line features is shown, comprising the following steps:

[0056] S1): Obtain the target model straight line, that is, select the key straight line in the target model as the model straight line, and the model straight line is represented by the position table of the two end points of the straight line in the body coordinate system;

[0057] S2): Acquire the target image to extract the straight line, that is, capture the target image by a camera, extract the straight line from the target area of ​​the image, and obtain the image straight line;

[0058] S3): Establish an intelligent agent, set the relative position and initial posture of the virtual target, as well as the initial action, and set the initial value of the action;

[0059] S4): Use the current position and posture of the virtual target to establish a distance matrix and observe its status;

[0060] S5): Calculate the current state reward value. If the reward value reaches the set threshold, output the target relative posture and straight line matching relationship; if the preset number of learning times is reached, the process ends;

[0061] S6): If the reward value in S5) does not reach the set threshold, the critic network is trained with the new reward value and Q value, and the actor network weight is updated;

[0062] S7): Get the next action through the actor network, update the position and posture, and return to step 4).

[0063] The deep reinforcement learning method for relative pose estimation of spatial targets based on straight line features described in this embodiment first determines the main straight line (called model straight line) information in the target model. After using a camera to obtain the target image, the target area in the image plane is extracted, and the extracted straight line is called a plane straight line. The distance between the projection of the model straight line on the plane and the plane straight line constitutes a distance matrix, which represents the observation space, and the state space is composed of all possible positions and postures of the target. The estimated value of the target pose is used as the initial state value, and the position and rotation direction of the agent relative to the current moment constitute the action space, and the reward value of each step is the distance between the model straight line in the image plane projection and the corresponding plane straight line. In this way, after the reinforcement learning process is completed, the distance between the model straight line in the image plane projection and the corresponding plane straight line can be minimized, thereby obtaining the corresponding relationship between the straight lines and the optimal pose parameter estimation value.

[0064] In the deep reinforcement learning method for estimating relative position and posture of a space target based on straight line features described in this embodiment, an intelligent agent is established in S3, and the initial values ​​of the relative position and posture of the target and the initial action are set, and the initial value of the action is set, as follows:

[0065] S0={rot0[3],tran0[3]} (1).

[0066] The deep reinforcement learning method for relative position and posture estimation of space targets based on straight line features described in this embodiment, the specific process of establishing a distance matrix using the current position and posture of the virtual target in S4) is as follows:

[0067] S41): First, solve the model straight line projection corresponding to the virtual target in the current position:

[0068] The model line is represented by two endpoints. Let P be the endpoint of the model line, and the target's posture and position relative to the camera coordinate system are R and T respectively. Then the position of P in the camera coordinate system can be expressed as:

[0069]

[0070] The projection point of P on the image plane is p = (p u ,p v ) can be expressed as:

[0071]

[0072] Among them, f x , f y , c x , c x are the focal length and optical center position of the camera in the x and y directions respectively;

[0073] S42): Calculate the distance between the model line projection and the plane line:

[0074] Known target model 3D straight line {L i , i = 1, 2, ..., m}, image plane straight line {l j ,j=1,2,...,n}, straight line L i It is represented by two endpoints The projections of the two endpoints on the image plane are p1 and i , p2 i ,

[0075]

[0076] Define the distance d between the model projection line i and the detection line j in the image plane ij is the model line L i The distances between the projection points p1 and p2 of the two endpoints on the image plane and the detection line l are represented by s1 and s2 respectively:

[0077] d ij =0.5*(s1+s2) (5);

[0078] S43): Establish a distance matrix:

[0079] The distance matrix between the model line and the plane line is an mxn dimensional matrix, where m is

[0080] The number of model lines, n is the number of plane lines, m>4, n>4;

[0081] O=D={d ij ,i=1,2,...,m,j=1,2,...,n} (6)

[0082] Among them, d ij Represents the distance between model line i and plane line j.

[0083] In the deep reinforcement learning method for relative pose estimation of space targets based on straight line features described in this embodiment, in the process of solving the distance between the model straight line projection and the plane straight line in S42), due to the discontinuity and elongation of the straight line extracted from the image, the image straight line l j It can be expressed as the equation of a plane line:

[0084] a j x+b j y+c j =0 (7)

[0085] A point p(x c ,y c ) to the plane line l j The distance s can be expressed as:

[0086] s=|a j x c +b j y c +c j | (8).

[0087] In the deep reinforcement learning method for relative pose estimation of space targets based on straight line features described in this embodiment, the current state reward value is calculated in S5), and the reward value for each step is the sum of each row in the observation matrix after executing the step, except for the minimum value selected, that is,

[0088]

[0089] match(k) represents the column number of the minimum value of the k-th row of matrix O;

[0090] At this time, if the reward value exceeds the set threshold, the target position and posture estimation values ​​are output;

[0091] If the exit condition of the reinforcement learning process is met, the process ends.

[0092] The deep reinforcement learning method for relative position and posture estimation of space targets based on straight line features described in this embodiment, S7): obtaining the next action through the actor network, updating the position and posture, is as follows:

[0093] S'=S+action(10).

[0094] Example 2

[0095] The deep reinforcement learning method for relative pose estimation of space targets based on straight line features described in this embodiment is the same as the method in Example 1, and is described below in conjunction with a specific embodiment.

[0096] A deep reinforcement learning method for relative pose estimation of spatial targets based on straight line features

[0097] The actual operation is as follows:

[0098] 1) Establish the DDPG network structure: a fully connected neural network of [16; 400; 600; 6], the network structure represents the number of nodes corresponding to the input layer, two hidden layers and the output layer. The hidden layers of the network use the RELU function as the activation function, and the output layer uses Tanh as the activation function. Batch_size = 64, the Actor network learning rate is 1e-5, the Critic network learning rate is 1e-4, and τ = 1e-3;

[0099] 2) Use opengl to create a cube with a side length of 10m. Its coordinates are defined as follows Figure 4 shown.

[0100] 3) Select 6 target model lines, each line is represented by two endpoints in the body coordinate system. The 6 lines are:

[0101] L1: [0,0,0,0,0,10],

[0102] L2:[0,0,10,10,0,10],

[0103] L3:[0,0,0,10,0,0],

[0104] L4:[10,0,10,10,0,0],

[0105] L5:[0,0,0,0,10,0],

[0106] L6:[0,10,0,10,10,0]

[0107] 4) Randomly rotate the object between 0-45 degrees to generate the target image;

[0108] 5) Obtain the target image for straight line extraction;

[0109] 6) Set the target relative position to [0.0.0.0.0.100.] and the initial action to [0.0.0.0.0.0.];

[0110] 7) Establish a distance matrix using the target’s current position and posture;

[0111] Assume P is the endpoint of the model line, and the target's relative posture and position to the camera coordinate system are R and T respectively. Then the position of P in the camera coordinate system can be calculated by the following formula:

[0112]

[0113] P c The projection point on the image plane is p=p(x c ,y c ) can be calculated by the following formula:

[0114]

[0115] Among them, f x ,f y ,c x ,c x are the focal length and optical center position of the camera in the x and y directions respectively.

[0116] Plane straight line l j The straight line equation is fitted as:

[0117] a j x c +b j y c +c j =0

[0118] Then calculate the model line projection endpoint p1(x c1 ,y c1 ) and p2(x c2 ,y c2 ) to the plane line l j The distances s1 and s2 are:

[0119] s1=|a j x c1 +b j y c1 +c j |

[0120] s2=|a j x c2 +b j y c2 +c j |

[0121] Model line i and plane line l j Distance:

[0122] d ij =0.5*(s1+s2)

[0123] Calculate the distance between all model lines and all plane lines to get the distance matrix (representing the observation state):

[0124] O=D={d ij ,i=1,2,...,m,j=1,2,...,n}

[0125] 8) Calculate the reward value under the current state

[0126] Reward value calculation formula:

[0127]

[0128] match(k) represents the column number of the minimum value in the k-th row of matrix O.

[0129] If the reward value reaches the set threshold, output the position and posture, and go to step 6); if the training rounds reach the predefined number, end the training.

[0130] Otherwise, the critic network is trained with the new reward and Q value, and the actor network weights are updated.

[0131] 9) Get the next action through the actor network, update the current posture state, and go to step 7).

[0132] After 400 rounds of training for 10 randomly generated initial values ​​(see Table 1), the reward function change trend is shown in the following figure: Figure 5 As shown, Figure 6 Shows snapshots of the agent's progress toward the true goal during training.

[0133] The final estimation result error is shown in Table 2.

[0134] Table 1. Relative poses of targets corresponding to the 10 images.

[0135]

[0136] Among them, Pitch, Roll, and Yaw represent pitch, roll, and yaw angles respectively, and Tx, Ty, and Tz represent the relative positions in the x, y, and z directions respectively.

[0137] Table 2 Pose estimation error obtained after reinforcement learning

[0138]

[0139] Among them, ΔPitch, ΔRoll, Δyaw represent the pitch, roll, and yaw angle errors respectively, and ΔTx, ΔTy, ΔTz represent the position errors in the x, y, and z directions respectively.

[0140] It should be noted that the DDPG algorithm is a general method of reinforcement learning and will not be introduced in detail here.

[0141] The above description is only a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements without departing from the principle of the present invention, and these improvements should also be regarded as within the protection scope of the present invention.

Claims

1. A deep reinforcement learning method for relative pose estimation of spatial targets based on straight line features. Features: The following steps are involved: S1): Obtain the target model straight line, that is, select the key straight line in the target model as the model straight line, and the model straight line is represented by the position table of the two end points of the straight line in the body coordinate system; S2): Acquire the target image to extract the straight line, that is, capture the target image by a camera, extract the straight line from the target area of ​​the image, and obtain the image straight line; S3): Establish an intelligent agent, set the relative position and initial posture of the virtual target, as well as the initial action, and set the initial value of the action; S4): Use the current position and posture of the virtual target to establish a distance matrix and observe its status; S5): Calculate the current state reward value. If the reward value reaches the set threshold, output the target relative posture and straight line matching relationship; if the preset number of learning times is reached, the process ends; S6): If the reward value in S5) does not reach the set threshold, the critic network is trained with the new reward value and Q value, and the actor network weight is updated; S7): Get the next action through the actor network, update the position and posture, and return to step 4).

2. According to the deep reinforcement learning method for relative pose estimation of space targets based on straight line features in claim 1, Features: In S3), an intelligent agent is established, and the target relative position and initial posture values, as well as the initial action, are set, and the initial action value is set, as follows: S 0 ={rot 0 [3],trans 0 [3]} (1)。 3. The deep reinforcement learning method for relative pose estimation of space targets based on straight line features according to claim 1, Features: The specific process of establishing the distance matrix using the current position and posture of the virtual target in S4) is as follows: S41): First, solve the model straight line projection corresponding to the virtual target in the current position: The model line is represented by two endpoints. Let P be the endpoint of the model line, and the target's posture and position relative to the camera coordinate system are R and T respectively. Then the position of P in the camera coordinate system can be expressed as: The projection point of P on the image plane is p = (p u ,p v ) can be expressed as: Among them, f x ,f y ,c x ,c x are the focal length and optical center position of the camera in the x and y directions respectively; S42): Solve the distance between the model line projection and the plane line: Known target model 3D straight line {L i , i = 1, 2, ..., m}, image plane straight line, {l j , j = 1, 2, ..., n} straight line L i It is represented by two endpoints (P 1 i,P 2 i), the projections of the two endpoints on the image plane are p 1 i ,p 2 i , Define the distance d between the model projection line i and the detection line j in the image plane ij is the model line L i The distances between the projection points p1 and p2 of the two endpoints on the image plane and the detection line l are expressed as s 1 ,s 2 express: d ij =0.5*(s 1 +s 2 ) (5); S43): Calculate the distances between all model lines and all plane lines, and establish a distance matrix: The distance matrix between the model lines and the plane lines is an m×n dimensional matrix, where m is the number of model lines and n is the number of plane lines, m>4, n>4; O=D={d ij ,i=1,2,...,m,j=1,2,...,n} (6) Among them, d ij Represents the distance between model line i and plane line j.

4. The deep reinforcement learning method for relative pose estimation of space targets based on straight line features according to claim 3, Features: In the process of solving the distance between the model straight line projection and the plane straight line in S42), due to the discontinuity and elongation of the straight line extracted from the image, the image straight line l j It can be expressed as the equation of a plane line: a j x+b j y+c j =0 (7) A point p(x c ,y c ) to the plane line l j The distance s can be expressed as: s=|a j x c +b j y c +c j | (8)。 5. The deep reinforcement learning method for relative pose estimation of space targets based on straight line features according to claim 3, Features: The current state reward value is calculated in S5), where the reward value for each step is the sum of each row in the observation matrix after executing the step, except for the minimum value that has been selected, that is, match(k) represents the column number of the minimum value of the k-th row of matrix O; At this time, if the reward value exceeds the set threshold, the target position and posture estimation values ​​are output; If the exit condition of the reinforcement learning process is met, the process ends.

6. The deep reinforcement learning method for relative pose estimation of space targets based on straight line features according to claim 1, Features: S7): Get the next action through the actor network and update the position and posture, as follows: S'=S+action(10).