A redundancy space robot arm null space obstacle avoidance motion planning method based on reinforcement learning

Abstract

The embodiment of the application provides a redundant space manipulator zero space obstacle avoidance motion planning method based on reinforcement learning, and belongs to the field of space manipulator motion planning, and comprises the following steps: obtaining an end velocity vector and a zero space motion vector according to the redundancy characteristics and free-floating characteristics of a redundant space manipulator; constructing a distance vector for describing the relative position between the redundant space manipulator and an obstacle; constructing state variables and action variables of the redundant space manipulator according to the distance vector and the zero space motion vector; constructing an obstacle avoidance reward function of the redundant space manipulator according to the distance vector and the zero space motion vector; constructing a zero space obstacle avoidance motion training strategy according to the state variables, the action variables and the obstacle avoidance reward function, and obtaining a trained zero space obstacle avoidance motion planning strategy network of the redundant space manipulator. According to the technical scheme provided by the embodiment of the application, the obstacle avoidance of the redundant space manipulator can be realized without changing the end position or attitude.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of space manipulator motion planning, and relates to a redundancy space manipulator null space obstacle avoidance motion planning method based on reinforcement learning. BACKGROUND

[0002] With the development of space technology, space exploration activities are becoming more and more frequent, and on-orbit operations such as spacecraft assembly, maintenance and debris removal are in urgent need. The space environment has characteristics such as no gravity, no air and high radiation, which makes the on-orbit operation of astronauts face great safety risks. As a typical space equipment, the space manipulator has high autonomy and flexibility, so it can replace astronauts to complete various on-orbit operation tasks.

[0003] Moving or carrying the load safely to the target position is a basic operation in on-orbit operation and is also a basic function of the space manipulator. Generally, the space manipulator is composed of a floating base, a multi-joint manipulator and a terminal for grabbing the load. Therefore, to realize the safe movement of the load, the following two conditions need to be met: first, a smooth and collision-free trajectory is planned so that the load can reach the target position along the trajectory; second, the base, the link and the load of the space manipulator need to avoid obstacles in the environment during movement. The method for realizing the above conditions is called space manipulator obstacle avoidance motion planning method. At present, most space manipulators have redundant degrees of freedom, which can realize zero-space obstacle avoidance without changing the end pose. However, there is a motion coupling between the base and the link of the space manipulator, and once the driving joint rotates, the base and the end will also produce motion at the same time, which increases the difficulty of obstacle avoidance of the space manipulator.

[0004] Current obstacle avoidance motion planning methods for redundant space manipulators mainly include gradient projection method and heuristic optimization method, etc. However, they introduce complex solving strategies and operation rules such as calculus and nonlinear functions, and have low calculation efficiency. In recent years, the development of reinforcement learning theory provides a new idea for solving the obstacle avoidance motion planning problem. Reinforcement learning constructs an intelligent agent that constantly interacts with the environment, and optimizes its action generation strategy according to the feedback state variables and rewards of the environment in each interaction process to maximize the total reward. It has good environmental adaptability and calculation real-time performance, and shows great potential. However, the existing motion planning methods based on reinforcement learning pay little attention to the obstacle avoidance problem in complex environment, which limits their application in the motion planning of redundant space manipulators. SUMMARY

[0005] Therefore, the application provides a redundancy space manipulator null space obstacle avoidance motion planning method based on reinforcement learning, so that the redundant space manipulator can efficiently avoid obstacles in the space environment without changing the end pose.

[0006] The application provides a redundancy space robot arm null space obstacle avoidance motion planning method based on reinforcement learning, comprising:

[0007] According to the redundancy characteristics and base free-floating characteristics of the redundancy space robot arm, the joint angular velocity vector of the redundancy space robot arm is decomposed to obtain an end velocity vector and a null space motion vector;

[0008] A distance vector for describing the relative position between the redundancy space robot arm and the obstacle is constructed;

[0009] According to the joint angle vector, the joint angular velocity vector, the distance vector and the null space motion vector of the redundancy space robot arm, a state variable and an action variable of the redundancy space robot arm are constructed;

[0010] According to the distance vector and the null space motion vector, an obstacle avoidance reward function of the redundancy space robot arm is constructed;

[0011] According to the state variable, the action variable and the obstacle avoidance reward function of the redundancy space robot arm, a null space obstacle avoidance motion training strategy is constructed to obtain a trained null space obstacle avoidance motion planning strategy network of the redundancy space robot arm.

[0012] In the above method, according to the redundancy characteristics and base free-floating characteristics of the redundancy space robot arm, the joint angular velocity vector of the redundancy space robot arm is decomposed to obtain an end velocity vector and a null space motion vector, comprising:

[0013] The joint angular velocity vector of the redundancy space robot arm is decomposed into:

[0014]

[0015] wherein, is the joint angular velocity vector, is the end velocity vector, is the null space motion vector, I is a unit matrix, and J f is a Jacobian matrix for describing the mapping relationship between the joint angular velocity vector and the end velocity vector in the base free-floating state, and

[0016] J f = J b J bm + J m

[0017] wherein, J b is a Jacobian matrix for describing the mapping relationship between the base velocity vector and the end velocity vector, J bm is a Jacobian matrix for describing the mapping relationship between the joint angular velocity vector and the base velocity vector, and J mA Jacobian matrix for describing the mapping relationship between the joint angular velocity vector and the end velocity vector of the redundant spatial manipulator in the fixed base state, J f Moore-Penrose pseudo-inverse.

[0018] In the method, the distance vector for describing the relative position between the redundant spatial manipulator and the obstacle is constructed, and the distance vector comprises:

[0019] Let the number of links of the redundant spatial manipulator be n, the number of components in the redundant spatial manipulator be i, i=0 represents the base, i=1~n respectively represents link 1~link n, the number of obstacles be k, and the number of obstacles be j, then the distance vector for describing the relative position between the component i of the redundant spatial manipulator and the obstacle is:

[0020]

[0021] wherein, is the position vector of a surface point on the component i, P i r is a set composed of the position vectors of all surface points on the component i; is the position vector of a surface point on the obstacle j, P is a set composed of the position vectors of all surface points on the obstacle j.

[0022] In the method, the state variable and the action variable of the redundant spatial manipulator are constructed according to the joint angle vector, the joint angular velocity vector, the distance vector and the null space motion vector of the redundant spatial manipulator, and the state variable and the action variable of the redundant spatial manipulator comprise:

[0023] (1) the state variable of the redundant spatial manipulator at time t is constructed as:

[0024]

[0025] wherein, q m and respectively represent the joint angle vector and the joint angular velocity vector at time t, p0, p1~pn respectively represent the distance vectors of the relative positions between the base, link 1~link n and the obstacles; n

[0026] (2) the action variable of the redundant spatial manipulator at time t is constructed as:

[0027]

[0028] wherein, is the null space motion vector of the redundant spatial manipulator.

[0029] ​The method comprises the following steps of:

[0030] The obstacle avoidance reward function of the redundant spatial manipulator at the t time is constructed as follows:

[0031] r t = λ1r o + λ2r a + f(p i )r c

[0032] Wherein, p i is a distance vector of the relative position between the spatial manipulator component i and the obstacle; r o is an obstacle avoidance feedback term, and its expression is as follows:

[0033]

[0034] Wherein, d s is a safety distance between the redundant spatial manipulator and the obstacle; r a is a motion stability term, and its expression is as follows:

[0035]

[0036] Wherein, is a null space motion vector of the redundant spatial manipulator, I is a unit matrix, J f is a Jacobian matrix describing the mapping relationship between the joint angular velocity vector and the end velocity vector in the free floating state of the base, is the Moore-Penrose pseudo-inverse of J f , λ1 and λ2 are weights of r o and r a respectively, r c is a collision penalty term, and r c =-1000; f(p i ) is an indication function for judging whether a collision occurs, and its expression is as follows:

[0037]

[0038] The method comprises the following steps of:

[0039] (1) The network structure required for training the redundant spatial manipulator null space obstacle avoidance motion is constructed according to the SAC algorithm, which comprises a strategy network π θ , two main Q networks and Two target Q networks And Wherein, the policy network π θ takes the state variable of the redundant space manipulator as input and takes the action variable of the redundant space manipulator as output; the main Q network and the target Q network both take the state variable and the action variable of the redundant space manipulator as input and take the Q value corresponding to the state variable and the action variable as output;

[0040] (2) The redundant space manipulator null space obstacle avoidance motion network is trained according to the following flow:

[0041] Step 1: specify the initial configuration q m0 of the redundant space manipulator, the number of obstacles k, the safety distance d s of the redundant space manipulator and the obstacle, the cumulative reward threshold R s , the time step Δt, the maximum time t max of each round in the training process, and the maximum number of rounds n ep_max ;

[0042] Step 2: create a data resource pool and initialize the policy network π θ , the main Q network and , and the target Q network and

[0043] Step 3: let the current round number n ep = 1;

[0044] Step 4: randomly select the positions of k obstacles that satisfy the condition 0 < ||p i || < d s (i = 1 ~ n, where n is the number of links of the redundant space manipulator);

[0045] Step 5: let the current time t = 0;

[0046] Step 6: input the state variable s t of the redundant space manipulator at the current time into the policy network π θ to obtain the action variable a t of the redundant space manipulator;

[0047] Step 7: drive the redundant space manipulator according to a t to obtain the state variable s t+1 at time t+1 and the obstacle avoidance reward function value r t+1 , and save them to the data resource pool;

[0048] Step 8: Sample training data from the data resource pool according to the SAC algorithm, and update the main Q network and target Q network and and the policy network π θ network parameters;

[0049] Step 9: Let t <- t + Δt;

[0050] Step 10: If t < t max , return to Step 6, otherwise skip this step;

[0051] Step 11: Calculate the cumulative reward

[0052] Step 12: If R > R s , let q m0 <- q m0 + Δq m0 , where Δq m0 is a random value that changes the initial configuration of the redundant space manipulator, otherwise skip this step;

[0053] Step 13: Let n ep <- n ep + 1;

[0054] Step 14: If n ep < n ep_max , return to Step 4, otherwise skip this step;

[0055] Step 15: End the training, and obtain the trained redundant space manipulator null space obstacle avoidance motion planning policy network π θ . BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative and laborious efforts.

[0057] Figure 1 is a flowchart of the redundant space manipulator null space obstacle avoidance motion planning method based on reinforcement learning provided by the embodiments of the present application;

[0058] Figure 2 is a schematic diagram of a planar three-degree-of-freedom redundant space manipulator model used in the simulation experiment in the embodiments of the present application;

[0059] Figure 3is a cumulative reward curve recorded by testing the policy network in the training process in the embodiment of the present application;

[0060] Figure 4 is an obstacle avoidance motion trajectory diagram of the redundant space robot base, link and end recorded by testing the policy network after training in the embodiment of the present application.

Specific embodiments

[0061] In order to better understand the technical solutions of the present application, the embodiments of the present application will be described in detail below with reference to the drawings.

[0062] It should be clear that the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0063] The embodiment of the present application provides a redundant space robot null space obstacle avoidance motion planning method based on reinforcement learning, please refer to Figure 1 , which is a flowchart of the redundant space robot null space obstacle avoidance motion planning method based on reinforcement learning provided by the present application, as shown in Figure 1 , the method comprises the following steps:

[0064] Step 101, according to the redundancy characteristics and base free floating characteristics of the redundant space robot, the joint angular velocity vector of the redundant space robot is decomposed to obtain the end velocity vector and the null space motion vector.

[0065] Specifically, according to the base free floating characteristics of the redundant space robot, the end velocity vector can be expressed as:

[0066]

[0067] wherein, is the end velocity vector, is the base velocity vector, is the joint angular velocity vector, J b is the Jacobian matrix describing the mapping relationship between the base velocity vector and the end velocity vector, J m is the Jacobian matrix describing the mapping relationship between the joint angular velocity vector and the end velocity vector in the base fixed state;

[0068] According to the relationship between the base velocity vector and the joint angular velocity vector of the redundant space robot:

[0069]

[0070] wherein, J bmA Jacobian matrix is used to describe the mapping relationship between the joint angular velocity vector and the base velocity vector, and the relationship between the end velocity vector of the redundant spatial manipulator and the joint angular velocity vector in the free-floating state of the base is obtained as follows:

[0071]

[0072] wherein J f is a Jacobian matrix used to describe the mapping relationship between the end velocity vector of the redundant spatial manipulator and the joint angular velocity vector in the free-floating state of the base, and

[0073] J f = J b J bm + J m (4)

[0074] According to the redundancy characteristic of the redundant spatial manipulator, the joint angular velocity vector of the redundant spatial manipulator is decomposed according to formula (3) as follows:

[0075]

[0076] wherein is a null space motion vector, I is a unit matrix, is the Moore-Penrose pseudo-inverse of J f .

[0077] Step 102, a distance vector used to describe the relative position between the redundant spatial manipulator and the obstacle is constructed.

[0078] Specifically, the number of links of the redundant spatial manipulator is denoted as n, the number of components in the redundant spatial manipulator is denoted as i, i=0 represents the base, i=1~n respectively represents link 1~link n, the number of obstacles is denoted as k, and the number of obstacles is denoted as j. Then, the distance vector of the relative position between the component i of the redundant spatial manipulator and the obstacle is constructed as follows:

[0079]

[0080] wherein is the position vector of a surface point on the component i, P i r is a set composed of the position vectors of all surface points on the component i; is the position vector of a surface point on the obstacle j, P is a set composed of the position vectors of all surface points on the obstacle j.

[0081] Step 103, according to the joint angle vector, the joint angular velocity vector, the distance vector and the null space motion vector of the redundant spatial manipulator, the state variable and the action variable of the redundant spatial manipulator are constructed.

[0082] Specifically, first, the state variable of the redundant spatial manipulator at time t is constructed as:

[0083]

[0084] wherein q m and respectively represent the joint angle vector and the joint angular velocity vector at time t, p0, p1~pn n respectively represent the distance vector of the relative position between the base, the link 1~the link n and the obstacle;

[0085] Secondly, the action variable of the redundant spatial manipulator at time t is constructed as:

[0086]

[0087] wherein, is the null space motion vector of the redundant spatial manipulator.

[0088] Step 104, according to the distance vector and the null space motion vector, the obstacle avoidance reward function of the redundant spatial manipulator is constructed.

[0089] Specifically, the obstacle avoidance reward function of the redundant spatial manipulator at time t is constructed as:

[0090] r t = λ1r o + λ2r a + f(p i )r c (9)

[0091] wherein p i is the distance vector of the relative position between the spatial manipulator component i and the obstacle; r o is the obstacle avoidance feedback term, and its expression is:

[0092]

[0093] wherein d s is the safety distance between the redundant spatial manipulator and the obstacle; r a is the motion stability term, and its expression is:

[0094]

[0095] wherein, is the null space motion vector of the redundant spatial manipulator, I is the unit matrix, J f is the Jacobian matrix describing the mapping relationship between the joint angular velocity vector and the end velocity vector in the free floating state of the base, is the J fMoore-Penrose pseudoinverse; λ1 and λ2 are r o and r a The weights; r c Let r be the collision penalty term. c =-1000; f(p) i The function that indicates whether a collision has occurred is expressed as follows:

[0096]

[0097] Step 105: Based on the state variables, action variables, and obstacle avoidance reward function of the redundant space manipulator, construct a zero-space obstacle avoidance motion training strategy to obtain the trained redundant space manipulator zero-space obstacle avoidance motion planning strategy network.

[0098] Specifically, firstly, the network structure required for training the zero-space obstacle avoidance motion of the redundant space robotic arm is constructed based on the SAC algorithm, which includes a policy network π. θ Two main Q networks and Two-target Q-network and Among them, the policy network π θ The state variables of the redundant space manipulator are used as input, and the motion variables of the redundant space manipulator are used as output; main Q-network and the target Q network Both take the state variables and motion variables of the redundant space robotic arm as inputs and output the Q value corresponding to the state variables and motion variables.

[0099] Secondly, the zero-space obstacle avoidance motion network of the redundant space robotic arm is trained according to the following procedure:

[0100] Step 1: Specify the initial configuration q of the redundant space robotic arm m0 Number of obstacles k, redundancy space, safe distance d between the robotic arm and obstacles s Cumulative reward threshold R s Time step Δt, the longest duration t of each round during training. max and the maximum number of rounds n ep_max ;

[0101] Step 2: Create a data resource pool and initialize the policy network π. θ Main Q network and and the target Q network and

[0102] Step 3: Let the current round number be n. ep =1;

[0103] Step 4: Randomly select a condition 0 < ||p i || < d s (i = 1 ~ n, where n is the number of links of the redundant spatial manipulator) of the positions of the k obstacles;

[0104] Step 5: Let the current time t = 0;

[0105] Step 6: Obtain the state variable s t of the redundant spatial manipulator at the current time, and input the policy network π θ , to obtain the action variable a t of the redundant spatial manipulator;

[0106] Step 7: Drive the redundant spatial manipulator according to a t , obtain the state variable s t+1 at time t + 1 and the obstacle avoidance reward function value r t+1 , and save them to the data resource pool;

[0107] Step 8: According to the SAC algorithm, sample training data from the data resource pool, and update the network parameters of the main Q network Q and the target Q network Q and and the policy network π θ ;

[0108] Step 9: Let t <— t + Δt;

[0109] Step 10: If t ≤ t max , return to Step 6, otherwise skip this step;

[0110] Step 11: Calculate the cumulative reward R

[0111] Step 12: If R > R s , let q m0 <— q m0 + Δq m0 , where Δq m0 is a random value that changes the initial configuration of the redundant spatial manipulator, otherwise skip this step;

[0112] Step 13: Let n ep <— n ep + 1;

[0113] Step 14: If n ep < n ep_max , return to Step 4, otherwise skip this step;

[0114] Step 15: end the training, and obtain the trained redundant space robot null space obstacle avoidance motion planning strategy network π θ .

[0115] According to the above method provided by the embodiment of the application, simulation experiments are performed on a planar three-degree-of-freedom redundant space robot with a floating base. Please refer to Figure 2 , which is a planar three-degree-of-freedom redundant space robot model, which only constrains the positions of the end in the x direction and the y direction under the inertial system I, and does not constrain the attitude of the end under the inertial system, so the end has 3 degrees of freedom and 2 constraints, that is, 1 redundant degree of freedom; and in the figure, I, b, 0, 1, 2, 3, and e represent the inertial system, the base coordinate system, the link 0 coordinate system, the link 1 coordinate system, the link 2 coordinate system, the link 3 coordinate system, and the end coordinate system, respectively, x I represents the unit vector of the X axis of the inertial system I, and y I represents the unit vector of the Y axis under the inertial system I, which is similar to the symbol. The D-H parameters of the redundant space robot are shown in Table 1, and the dynamics parameters are shown in Table 2.

[0116] Table 1 D-H parameters of the redundant space robot

[0117]

[0118] Table 2 Dynamics parameters of the redundant space robot

[0119]

[0120]

[0121] Table 3 Training process parameter values

[0122]

[0123] In the PyGame module based on the OpenAI Gym framework, a simulation environment is built, the initial configuration q m0 of the redundant space robot is taken as [-45, -90, 45] T (°), other parameters used in the training are shown in Table 3, and the technical solution of the embodiment of the application is used for training. During the training, the strategy network π θ is tested every 4 rounds, and the cumulative reward obtained is recorded, please refer to Figure 3 . It can be seen from Figure 3 that the cumulative reward obtained gradually increases with the increase of the number of training rounds, indicating that the strategy network π θGradually learn to make redundant space manipulator to realize zero space obstacle avoidance operation;After training 300 rounds, the cumulative reward gradually reduces the amplitude of increase, and the fluctuation gradually decreases, indicating that the training process of the strategy network π θ Gradually completes.

[0124] Take out the trained strategy network π θ , place it in a simulation environment with an obstacle for testing, and the test results are shown in Figure 4 In Figure 4 , the base and link of the redundant space manipulator gradually transition from light to dark over time, so that the motion trajectory can be visually observed. At the initial moment, the distance between the obstacle and the link is less than the safety distance d s . Under the drive of the strategy network π θ , the redundant space manipulator moves away from the obstacle until it is greater than the safety distance d s , and then gradually stops moving. Therefore, the redundant space manipulator successfully avoids the obstacle, and the position of the end does not change during the process, thus proving the correctness and effectiveness of the redundant space manipulator zero space obstacle avoidance motion planning method based on reinforcement learning.

[0125] The technical scheme of the embodiment of the present application has the following beneficial effects:

[0126] By utilizing the zero space motion characteristics of the redundant space manipulator, introducing the reinforcement learning theory, constructing the state variables, action variables and reward functions of the zero space obstacle avoidance motion planning problem, and designing the zero space obstacle avoidance motion training strategy, a redundant space manipulator zero space obstacle avoidance motion planning method based on reinforcement learning is formed, which can effectively avoid obstacles without changing the position or attitude of the end, and has good adaptability to unstructured environments and sufficient computing efficiency.

[0127] The above only describes the preferred embodiments of the present application and does not limit the present application, any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0128] The contents not described in detail in the specification of the present application are well-known technologies to those skilled in the art.

Claims

1. A redundancy space robot arm null space obstacle avoidance motion planning method based on reinforcement learning, characterized in that, The method comprises: According to the redundancy characteristics and base free floating characteristics of the redundant space manipulator, the joint angular velocity vector of the redundant space manipulator is decomposed to obtain an end velocity vector and a null space motion vector; A distance vector for describing the relative position between the redundant space manipulator and the obstacle is constructed; According to the joint angle vector, the joint angular velocity vector, the distance vector and the null space motion vector of the redundant space manipulator, a state variable and an action variable of the redundant space manipulator are constructed; According to the distance vector and the null space motion vector, an obstacle avoidance reward function of the redundant space manipulator is constructed; According to the state variable, the action variable and the obstacle avoidance reward function of the redundant space manipulator, a null space obstacle avoidance motion training strategy is constructed to obtain a trained null space obstacle avoidance motion planning strategy network of the redundant space manipulator.

2. The method of claim 1, wherein, According to the redundancy characteristics and base free floating characteristics of the redundant space manipulator, the joint angular velocity vector of the redundant space manipulator is decomposed to obtain an end velocity vector and a null space motion vector, comprising: The joint angular velocity vector of the redundant space manipulator is decomposed into: wherein, is the joint angular velocity vector, is the end-effector velocity vector, is the null-space motion vector, I is the identity matrix, J f is the Jacobian matrix describing the mapping between the joint angular velocity vector and the end-effector velocity vector in the free-floating base state, and J f =J b J bm +J m where J b is the Jacobian matrix describing the mapping between the base velocity vector and the end-effector velocity vector, J bm is the Jacobian matrix describing the mapping between the joint angular velocity vector and the base velocity vector, J m is the Jacobian matrix describing the mapping between the joint angular velocity vector and the end-effector velocity vector in the base-fixed frame, is the Moore-Penrose pseudo-inverse of J f .

3. The method of claim 1, wherein, The distance vector for describing the relative position between the redundant space manipulator and the obstacle is constructed, comprising: Let the number of links of the redundant space manipulator be n, the number of components in the redundant space manipulator be i, i=0 represents the base, i=1~n respectively represents link 1~link n, the number of obstacles be k, and the number of obstacles be j, then the distance vector for describing the relative position between the redundant space manipulator component i and the obstacle is: wherein, P is the position vector of a certain surface point on component i, i r S is the set of position vectors of all surface points on component i; P is the position vector of a certain surface point on obstacle j, S is the set of position vectors of all surface points on obstacle j.

4. The method of claim 1, wherein, According to the joint angle vector, the joint angular velocity vector, the distance vector and the null space motion vector of the redundant space manipulator, a state variable and an action variable of the redundant space manipulator are constructed, comprising: (1) The state variable of the redundant space manipulator at time t is constructed as: where q m and respectively represent the joint angle vector and joint angular velocity vector at time t, p0, p1~pn n respectively represent the distance vectors of the relative positions between the base, the links 1~n and the obstacle. (2) The action variable of the redundant space manipulator at time t is constructed as: wherein, is the null space motion vector for the redundant spatial robot arm.

5. The method of claim 1, wherein, According to the distance vector and the null space motion vector, an obstacle avoidance reward function of the redundant space manipulator is constructed, comprising: The obstacle avoidance reward function of the redundant space manipulator at time t is constructed as: r t = λ1r o + λ2r a + f(p i )r c where p i is the distance vector of the relative position between the spatial manipulator assembly i and the obstacle; r o is the obstacle avoidance feedback term, whose expression is: where d s is the safety distance between the redundant spatial manipulator and the obstacle; r a is the motion stability term, which is expressed as: wherein, is the null space motion vector of the redundant spatial manipulator, I is an identity matrix, J f is the Jacobian matrix describing the mapping relationship between the joint angular velocity vector and the end velocity vector in the free-floating state of the base, is the Moore-Penrose pseudo-inverse of J f ; λ1 and λ2 are the weights of r o and r a , respectively; r c is a collision penalty term, and r c =-1000; f(p i ) is an indicator function for judging whether a collision occurs, and its expression is:

6. The method of claim 1, wherein, According to the state variable, the action variable and the obstacle avoidance reward function of the redundant space manipulator, a null space obstacle avoidance motion training strategy is constructed to obtain a trained null space obstacle avoidance motion planning strategy network of the redundant space manipulator, comprising: (1) Construct the network structure required for training the redundant space of the redundant space manipulator to avoid obstacles according to the SAC algorithm, which includes a policy network π θ , two main Q networks and two target Q networks and wherein the policy network π θ takes the state variables of the redundant space manipulator as input and takes the action variables of the redundant space manipulator as output; the main Q network and the target Q network both take the state variables and the action variables of the redundant space manipulator as input and take the Q values corresponding to the state variables and the action variables as output; (2) The redundant space manipulator null space obstacle avoidance motion network is trained according to the following process: Step 1: specify the initial configuration q of the redundant spatial manipulator m0 , the number of obstacles k, the safety distance d of the redundant spatial manipulator and the obstacle s , the cumulative reward threshold R s , the time step Δt, the maximum time t of each round in the training process max and the maximum number of rounds n ep_max ; Step 2: Create a data resource pool, initialize the policy network π θ , the main Q network and , and the target Q network and Step 3: Let n be the current round number ep = 1; Step 4: Randomly select a condition 0 < ||p i || < d s (i = 1 ~ n, where n is the number of links of the redundant spatial robot arm) of the k obstacles; Step 5: Let the current time t=0; Step 6: obtain the state variable s of the redundant spatial manipulator at the current time t input policy network π θ , and obtain the action variable a of the redundant spatial manipulator t ; Step 7: According to a t Drive the redundant spatial manipulator to obtain the state variable s at time t+1 t+1 And the obstacle avoidance reward function value r t+1 Save it to the data resource pool; Step 8: Sample training data from the data resource pool according to the SAC algorithm, update the main Q network and target Q network and and the policy network π θ network parameters; Step 9: Let t←t+Δt; Step 10: If t < t max , go to Step 6, otherwise skip this step; Step 11: Calculate cumulative rewards Step 12: If R > R s , let q m0 ← q m0 + Δq m0 , where Δq m0 is a random value that changes the initial configuration of the redundant spatial manipulator, otherwise skip this step; Step 13: Let n ep ← n ep + 1 ; Step 14: If n ep Step 13: If n ep_max , go to Step 4, otherwise skip this step; Step 15: End the training, and obtain the trained redundant space manipulator null space obstacle avoidance motion planning strategy network θ .

Images