Steam generator tube sheet quadruped robot landing point planning method and system

Abstract

The steam generator tube plate quadruped robot landing point planning method and system belong to the technical field of nuclear industry robot motion planning. In order to solve the problem that the existing steam generator tube plate quadruped robot landing point method needs a large number of complex analyses on the motion form, working condition and future possible situation of the robot, the motion space is constructed according to the landing action corresponding to the actual landing position of the robot, the state space is constructed according to the position and posture information R, the end position in the global motion direction and the judgment flag f of whether the landing point corresponding to the action space can actually land, the flag f(s) corresponding to the state s is used as the mask of the state s; meanwhile, the reward function is constructed by using the daily punishment, the reward / punishment of forward / backward motion and the stable reward / punishment; then the D3QN model is used for landing point planning, a layer of mask is added after the D3QN output Q value; so as to complete the landing point planning based on the D3QN.

Description

Technical Field

[0001] This invention belongs to the field of motion planning technology for nuclear industry robots, specifically relating to a method for planning the landing points of a quadruped robot for steam generator tube sheets. Background Technology

[0002] With the rapid development of nuclear energy, robotics research in the nuclear power field has become increasingly popular. Tube sheet quadruped robots can adapt to different models and specifications of tube sheets, moving flexibly and safely on steam generator tube sheets. Traditional foot placement planning methods for tube sheet quadruped robots can meet the task requirements of robot movement in small-scale environments. However, for movement tasks in certain specific environments, traditional planning methods require precise modeling and analysis of the robot's kinematics, as well as extensive complex analysis of the robot's motion patterns, working conditions, and possible future scenarios. The resulting motion plans may even be infeasible. Therefore, a more general and automated method is needed to implement quadruped robot motion planning. This involves specifying a high-level task, and the algorithm determining the robot's motion while considering the physical constraints of the robot's feet and base. Summary of the Invention

[0003] This invention aims to address the problem that existing methods for determining the landing point of a quadruped robot on a steam generator tube sheet require extensive and complex analysis of the robot's motion patterns, operating conditions, and potential future scenarios.

[0004] A method for planning the landing points of a quadruped robot on a steam generator tube sheet is provided. The method involves planning the landing points of the quadruped robot on the tube sheet. The quadruped robot includes a base module and four leg modules. The four leg modules are the robot's four legs, which are set on the base module. The D3QN model is used for planning the landing points of the quadruped robot on the tube sheet.

[0005] During the planning process, the motion space, state space, and reward function of the tube sheet quadruped robot are as follows:

[0006] Action space:

[0007]

[0008] in, For each landing action, the corresponding landing point for each action. It is relative to the current foot position The actual landing position is k max It is the number of landing points that satisfy all possible cases of robot inverse kinematics, and it is also the dimension of the action space;

[0009] State space:

[0010] s C=[R, Pos T f]

[0011] state space s C Includes the robot's position and orientation information R, and the endpoint Pos in the global motion direction. T (r T c T The criteria for determining whether a foot can actually be placed in the action space. Where r T c T The row and column coordinates of the endpoint in the global motion direction. f is a flag indicating whether a point in the robot's current action space can be placed. k =1 indicates that the point can be placed, f k =0 indicates that the point cannot be placed. The flag f(s) corresponding to state s constitutes the mask of state s.

[0012] Reward function:

[0013] R C =-m0+R foot +m1R stable

[0014] R C It consists of three parts with different weights. The first part is the daily penalty, which is a penalty of m0 for each step, to ensure that the robot does not move too much or stop in place. The second part is the reward / penalty for moving forward / backward. The third part is about stable rewards / penalties.

[0015] R foot_k and R stable_k as follows:

[0016]

[0017]

[0018] Among them, l foot_k This represents the distance the foot travels along the direction of movement; a value greater than 0 indicates approaching the target, a value less than 0 indicates moving away from the target, and 0 indicates the foot is stationary in the direction of movement. foot_max This indicates the maximum distance the foot can travel along the direction of movement within the movement space; l c_to_sup_k This represents the distance between the centroid of the stable triangle formed by the current supporting feet and the base; l c_to_line_k l represents the distance from the centroid of the stable triangle formed by the current supporting feet to the line connecting the start and end points. stable_maxThis represents the farthest distance from the centroid of the stable triangle formed by the current supporting feet to the side of the triangle; m1, m2, ..., m8 are the parameters of the reward function;

[0019] Based on the aforementioned action space, state space, and reward function, the action selection strategy for foot placement planning using the D3QN model is as follows:

[0020]

[0021] Among them, random action from feasible A f (s) represents the list of feasible landing points. In the selection of random actions, 0≤ξ≤1 is a uniformly random number drawn in each time step, 0<∈<1 is a fixed probability in ∈-greedy, and Q(s,a) is the Q value corresponding to the selection of action a in state s.

[0022] Furthermore, the state space s C Position and attitude information R = [r B c B ,θ1,θ 21 θ 31 θ 22 θ 32 θ 23 θ 33 θ 24 θ 34 ], where r B c B Here are the row and column coordinates of the robot's current position, θ1 is the rotation angle of the robot's base, and θ 21 θ 31 θ represents the two turning angles of the robot's left front leg. 22 θ 32 θ represents the two turning angles of the robot's left front leg. 23 θ 33 θ represents the two turning angles of the robot's left hind leg. 24 θ 34 These are the two corners of the robot's right hind leg.

[0023] Furthermore, in the process of foot placement planning using the D3QN model, the calculation formula for the target network in D3QN is as follows:

[0024]

[0025] Among them, y t Represents the target value at time t; γ is the discount factor, r t It is the reward function at time t, Q(s) t+1 a) is about state st+1 And the state-action value function of the action, corresponding to The Q value of the target network, To evaluate the Q-value of the network; "next state is final state" means that the next state is the final state.

[0026] Furthermore, in the process of foot placement planning using the D3QN model, the formula for evaluating the value of network-computed state actions in D3QN is as follows:

[0027]

[0028] in, It's about state s t The state value function, It's about state s t and action a t The dominance function, mean, represents the mean value, w, w V and w A These are the parameters for the corresponding network.

[0029] Furthermore, the neural network model corresponding to the evaluation network in D3QN adopts a fully connected neural network, which includes an input layer, two hidden layers of size 64, and an output layer processed using the sigmoid activation function.

[0030] A landing point planning system for a quadruped robot on a steam generator tube sheet is provided. The system plans the landing points of the quadruped robot on the tube sheet. The quadruped robot includes a base module and four leg modules. The four leg modules are the robot's four legs, which are set on the base module. The system includes a state information storage module and a planning module.

[0031] State information storage module: Used to store the motion space and state space information of the tube sheet quadruped robot.

[0032] Action space:

[0033]

[0034] in, For each landing action, the corresponding landing point for each action. It is relative to the current foot position The actual landing position is k max It is the number of landing points that satisfy all possible cases of robot inverse kinematics, and it is also the dimension of the action space;

[0035] State space:

[0036] sC =[R, Pos T f]

[0037] state space s C Includes the robot's position and orientation information R, and the endpoint Pos in the global motion direction. T (r T c T The criteria for determining whether a foot can actually be placed in the action space. Where r T c T The row and column coordinates of the endpoint in the global motion direction. f is a flag indicating whether a point in the robot's current action space can be placed. k =1 indicates that the point can be placed, f k =0 indicates that the point cannot be placed. The flag f(s) corresponding to state s constitutes the mask of state s.

[0038] Planning module: Reads the action space and state space from the state information storage module, and calls the D3QN model to plan the landing point. The reward function and action selection strategy used in the D3QN model's landing point planning process are as follows:

[0039] Reward function:

[0040] R C =-m0+R foot +m1R stable

[0041] R C It consists of three parts with different weights. The first part is the daily penalty, which is a penalty of m0 for each step, to ensure that the robot does not move too much or stop in place. The second part is the reward / penalty for moving forward / backward. The third part is about stable rewards / penalties.

[0042] R foot_k and R stable_k as follows:

[0043]

[0044]

[0045] Among them, l foot_k This represents the distance the foot travels along the direction of movement; a value greater than 0 indicates approaching the target, a value less than 0 indicates moving away from the target, and 0 indicates the foot is stationary in the direction of movement. foot_max This indicates the maximum distance the foot can travel along the direction of movement within the movement space; lc_to_sup_k This represents the distance between the centroid of the stable triangle formed by the current supporting feet and the base; l c_to_line_k l represents the distance from the centroid of the stable triangle formed by the current supporting feet to the line connecting the start and end points. stable_max This represents the farthest distance from the centroid of the stable triangle formed by the current supporting feet to the side of the triangle; m1, m2, ..., m8 are the parameters of the reward function;

[0046] Action selection strategy:

[0047]

[0048] Among them, random action from feasible A f (s) represents the list of feasible landing points. In the selection of random actions, 0≤ξ≤1 is a uniformly random number drawn in each time step, 0<∈<1 is a fixed probability in ∈-greedy, and Q(s,a) is the Q value corresponding to the selection of action a in state s.

[0049] Furthermore, the state space s C Position and attitude information R = [r B c B ,θ1,θ 21 θ 31 θ 22 θ 32 θ 23 θ 33 θ 24 θ 34 ], where r B c B Here are the row and column coordinates of the robot's current position, θ1 is the rotation angle of the robot's base, and θ 21 θ 31 θ represents the two turning angles of the robot's left front leg. 22 θ 32 θ represents the two turning angles of the robot's left front leg. 23 θ 33 θ represents the two turning angles of the robot's left hind leg. 24 θ 34 These are the two corners of the robot's right hind leg.

[0050] Furthermore, in the process of foot placement planning using the D3QN model, the calculation formula for the target network in D3QN is as follows:

[0051]

[0052] Among them, y t Represents the target value at time t; γ is the discount factor, rt It is the reward function at time t, Q(s) t+1 a) is about state s t+1 And the state-action value function of the action, corresponding to The Q value of the target network, To evaluate the Q-value of the network; "next state is final state" means that the next state is the final state.

[0053] Furthermore, in the process of foot placement planning using the D3QN model, the formula for evaluating the value of network-computed state actions in D3QN is as follows:

[0054]

[0055] in, It's about state s t The state value function, It's about state s t and action a t The dominance function, mean, represents the mean value, w, w V and w A These are the parameters for the corresponding network.

[0056] Furthermore, the neural network model corresponding to the evaluation network in D3QN adopts a fully connected neural network, which includes an input layer, two hidden layers of size 64, and an output layer processed using the sigmoid activation function.

[0057] Beneficial effects:

[0058] This invention uses MD3QN to plan the landing points of a quadruped robot for a steam generator tube sheet. This eliminates the need for extensive and complex analysis of the robot's motion patterns, working conditions, and potential future scenarios. Furthermore, the landing point planning of this invention can effectively determine feasible motion plans.

[0059] The evaporator tube sheet used in this invention can be simplified into a two-dimensional grid environment, divided into square and triangular distributions of tube holes. The final training results of the 10 environments and tasks corresponding to the planning using this invention are shown in Table 2. Under different environments and tasks, compared with the traditional D3QN algorithm, the landing point planning results of this invention have higher total scores, fewer steps, and better stability. The comparative experimental results prove the effectiveness of the D3QN_Mask algorithm. Attached Figure Description

[0060] Figure 1 The image in the center is a 3D structural diagram of a quadrupedal maintenance robot.

[0061] Figure 2This is a plan view of a quadrupedal maintenance robot with all four legs fixed to the tube sheet.

[0062] Figure 3 This is a top view of a quadrupedal maintenance robot with all four legs fixed to a tube sheet.

[0063] Figure 4 This is a schematic diagram of the two-dimensional grid environment of the evaporator tube sheet and the distribution of its tube holes.

[0064] Figure 5 The curves for average round reward and average round steps during the training process of Square-1 and Triangle-1 are shown.

[0065] Figure 6 This is a diagram illustrating a training task. Detailed Implementation

[0066] This invention proposes a gait planner based on the reinforcement learning MD3QN algorithm. By adding a mask during action selection, designing a reward function for the walking task, and combining the constraints of the foot and base motion space, it achieves autonomous learning of foot placement planning for a quadruped robot on a tube sheet. This enables the robot to move along the desired trajectory on the tube sheet. The invention is further described below with reference to specific embodiments. Specific implementation method one:

[0068] This embodiment is a landing point planning method for a quadruped robot on a steam generator tube sheet. It is a landing point planning method for a quadruped robot on a steam generator tube sheet based on MD3QN. MD3QN is a D3QN model with a mask added to the selection strategy. The D3QN algorithm is a variant algorithm that combines Double DQN and Dueling DQN. It can eliminate the maximization bias in the network update process, solve the problem of overestimation, and accelerate the convergence of the algorithm.

[0069] The method for planning the landing points of a quadruped robot on a steam generator tube sheet, as described in this embodiment, includes the following steps:

[0070] Step 1: Analyze the structure and motion characteristics of the tube sheet quadruped robot:

[0071] 1.1 Structure of the tube sheet quadruped robot:

[0072] The structure of a quadruped maintenance robot for overhauling heat transfer tubes of multi-specification steam generators is as follows: Figures 1-3 As shown, Figure 1 This is a 3D structural diagram of a quadrupedal maintenance robot. Figure 2 This is a plan view of a quadrupedal maintenance robot with all four legs fixed to the tube sheet. Figure 3 This is a top view of a quadrupedal maintenance robot with all four legs fixed to a tube sheet.

[0073] The quadrupedal maintenance robot includes a base module 10, a working arm module 30, four leg modules 20, and four toe modules 40.

[0074] Four leg modules are mounted on the base module; a passive rotary joint is located at the center of the base module, such as... Figure 1 In the rotation marker 1, each leg module includes a foot end and a rotational joint connecting the foot end to the base. The rotational joint connecting the foot end to the base is designated as the first rotational joint. The foot end has two rotational joints; the one closer to the first rotational joint is designated as the second rotational joint, and the other is designated as the third rotational joint. Figure 1 Rotation markers 2 and 3 in the diagram indicate that these three types of rotary joints determine the robot's movement in the plane; the combined relative motion of the base module and the four leg modules is used to enable the robot to walk on the tube sheet;

[0075] The working arm module is installed below the base module and carries eddy current tools to inspect the heat transfer tubes;

[0076] Each foot tip is equipped with a movable joint, such as Figure 1 The four toe modules, each located at the end of one of the four leg modules via movable joints, are positioned to lift and grip the tube sheet, then release and descend to detach from it. These toe modules ensure contact between the robot and the tube sheet, guaranteeing stable and reliable mounting even with three or four toes attached, without considering ZMP-like stabilization conditions. (The last sentence appears to be an unrelated analogy and is left untranslated.) Figure 1 As shown in (b) and (c), the motion space of the robot's feet and the motion space of its base can be obtained based on the robot's structure.

[0077] The robot's left front leg is defined as the lf leg, the left rear leg as the lb leg, the right front leg as the rf leg, and the right rear leg as the rb leg. During movement, each leg lands in the gait sequence of lf-rf-rb-lb.

[0078] 1.2 Motion characteristics of the tube sheet quadruped robot:

[0079] In terms of stability: Due to the special structure of the tube plate quadruped robot, as long as the toe module is fixed to the tube plate, the robot can be guaranteed to be stable and will not tip over or fall.

[0080] Regarding the selection of landing points: For the landing points of the tube plate quadruped robot, as long as the kinematic conditions are met, and the tube hole corresponding to the position is not an obstacle tube hole and there will be no collision, these positions are all feasible landing points.

[0081] Although the robot can be stably fixed in any posture on the tube sheet, the toe modules are subjected to different forces. Therefore, considering the service life of the parts, when planning the landing point for a given motion task, in addition to improving motion efficiency, the force on the toe modules should be minimized as much as possible. That is, the center of mass of the robot body should be as close as possible to the given base trajectory, and the center of mass should be as close as possible to or inside the supporting polygon.

[0082] Step 2: Determine the motion space and state space of the tube sheet quadruped robot during its movement.

[0083] Action space Most quadruped robots that apply reinforcement learning typically use joint positions as their motion space. Since each foot placement corresponds to a specific orifice, to reduce unnecessary exploration at the decision layer, the motion space of this invention is defined as the positions of the surrounding orifices relative to the current foot's position. This is a fixed motion space, as follows:

[0084]

[0085] in, For each landing action, the corresponding landing point for each action. It is relative to the current foot position The actual landing position is k max It represents the number of foot placement points that satisfy all possible conditions of inverse kinematics, and is also a dimension of the action space. Among these, each foot placement point corresponds to... This represents the row and column coordinates. For example, suppose the current foot position is (r... t c t ), r t c t Let a1 be the row and column coordinates of the current foot, then a1 represents the position where the current foot lands (r). t +1, c t +1) Operation of the tube hole.

[0086] state space state space s C as follows:

[0087] s C =[R, Pos T f]

[0088] state space s C Includes the robot's position and orientation information R = [r B c B ,θ1,θ 21 θ 31 θ 22 θ 32 θ 23θ 33 θ 24 θ 34 The endpoint Pos in the global motion direction T (r T c T The criteria for determining whether a foot can actually be placed in the action space. Where, r B c B Here are the row and column coordinates of the robot's current position, θ1 is the rotation angle of the robot's base, and θ 21 θ 31 θ represents the two turning angles of the robot's left front leg. 22 θ 32 θ represents the two turning angles of the robot's left front leg. 23 θ 33 θ represents the two turning angles of the robot's left hind leg. 24 θ 34 For the two corners of the robot's right hind leg, r T c T The row and column coordinates of the endpoint in the global motion direction. f is a flag indicating whether a point in the robot's current action space can be placed. k =1 indicates that the point can be placed, f k =0 indicates that the point cannot be placed.

[0089] Step 3: Define the reward function for the movement process of the tube sheet quadruped robot based on its motion characteristics:

[0090] reward function The reward function R designed in this invention C It consists of three parts with different weights: one part is the daily penalty, i.e., a penalty of m0 for each step, to ensure that the robot does not move too much or stop in place; another part is the reward / penalty for moving forward / backward. One part is stable rewards / penalties

[0091] R C Specifically as follows:

[0092] R C =-m0+R foot +m1R stable

[0093] Among them, R foot_k and R stable_k as follows:

[0094]

[0095]

[0096] Among them, l foot_k This represents the distance the foot travels along the direction of movement; a value greater than 0 indicates approaching the target, a value less than 0 indicates moving away from the target, and 0 indicates the foot is stationary in the direction of movement. foot_max This indicates the maximum distance the foot can travel along the direction of movement within the movement space; l c_to_sup_k This represents the distance between the centroid of the stable triangle formed by the current supporting feet and the base; l c_to_line_k l represents the distance from the centroid of the stable triangle formed by the current supporting feet to the line connecting the start and end points. stable_max This represents the farthest distance from the centroid of the stable triangle formed by the current supporting feet to a side of the triangle. m1, m2, ..., m8 are parameters of the reward function.

[0097] Step 4: Determine the termination conditions during the reinforcement learning training process:

[0098] Termination Condition: Initializing the reference state and prematurely terminating erroneous actions during training is necessary to avoid wasting computational resources on incorrect strategies. Therefore, this invention designs the following termination condition:

[0099] 1) Successful termination conditions:

[0100] The robot's base position is the same as the target base position, and it successfully reaches the destination.

[0101] 2) Termination conditions upon failure:

[0102] The robot's base position did not advance for several consecutive times, and the base position oscillated repeatedly. Repeated oscillation refers to repeated movement between two positions.

[0103] The swinging foot has no place to land.

[0104] Step 5: Design action selection strategies during the reinforcement learning algorithm training process:

[0105] Action Selection: How to choose actions essentially boils down to balancing exploration and exploitation, a problem often referred to as the exploration-exploitation dilemma. A simple, straightforward, and usually very successful approach is the ∈-greedy method. In the ∈-greedy method, at each time step, the agent chooses a random action with a fixed probability of 0 < ∈ < 1, rather than simply greedily selecting the learned optimal action with respect to the Q-function. The Q-network implemented using a neural network is the evaluation network in the MD3QN algorithm. The input layer of the neural network represents the states in the state space, and the output layer represents the Q-value of each action, i.e., the score of each action. In this implementation, a fully connected neural network is used, containing an input layer, two hidden layers of size 64, and an output layer processed using the sigmoid activation function. The final output of the neural network is the score of each action. For state s, during greedy selection, after the MD3QN algorithm outputs the Q value, a mask is added (the mask is composed of the landability flag f(s)). That is, each output value Q is multiplied by the input landability flag f(s), ensuring that the action with the highest score is always reachable, thus avoiding a large number of unnecessary failure and termination conditions. Similarly, during random selection, only the list of feasible landing points is considered. A random action is selected from the given options. The action selection strategy is as follows:

[0106]

[0107] Among them, random action from feasible A f (s) represents the list of feasible landing points. In the selection of random actions, 0≤ξ≤1 is a uniformly random number drawn in each time step, and Q(s,a) is the Q value corresponding to the selection of action a in state s.

[0108] Step 6: Design a gait planner for foot placement planning:

[0109] This is a sequential decision-making problem involving the interaction between an agent and its environment, with the objective of maximizing the cumulative reward (which is discounted). This problem can be modeled as a discrete-time Markov decision process (MDP), where the MDP consists of tuples. Composition, here γ represents the robot's state, action, state transition distribution, reward function, and discount coefficient, respectively.

[0110] During MD3QN training, as described above, the state The input to the neural network (evaluation network) is the Q-value of each action, which serves as the output of the neural network. Following the action selection method in step 5, the robot selects actions and interacts with the environment, causing a state transition. Simultaneously, the environment provides a reward function. The action selection strategy chooses the next action based on the new state and reward function. The process of continuously updating the policy network is the training process of the reinforcement learning algorithm.

[0111] MD3QN algorithm learning process:

[0112] After establishing the MD3QN neural network model, it is necessary to continuously iterate and update the parameters of the neural network. At this time, it is necessary to use the evaluation network and the target network to perform calculations to update the MD3QN neural network parameters.

[0113] D3QN offers two main optimizations compared to the traditional DQN algorithm. First, the TD error objective is optimized in line with the DoubleDQN algorithm, and the update utilizes both networks of DQN: the evaluation network (with parameter w) is used. e To determine the next action, use the target network (parameter w) t This is used to calculate the value of the state at time t+1, thereby reducing overestimation. The target of the target network during D3QN update is shown in the following formula.

[0114]

[0115] Among them, y t This represents the target value at time t, which is the calculated Q-value of the target network; γ is the discount factor, r t It is the reward function at time t, Q(s) t+1 a) is about state s t+1 And the state-action value function of the action, corresponding to The Q value of the target network, To evaluate the Q-value of the network; "next state is final state" means that the next state is the final state.

[0116] Secondly, the D3QN algorithm divides the state-action value function into two parts, consistent with the Dueling DQN algorithm. It models the state value function and the advantage function separately to better handle states with less correlation to actions. The newly created state-action value function is shown in the following equation. Here, the mean is used instead of the maximization operation for more stable operation.

[0117]

[0118] in, It's about state s tThe state value function, It's about state s t and action a t The dominance function, mean, represents the mean value, w, w V and w A These are the parameters for the corresponding network.

[0119] Based on the D3QN algorithm, this invention establishes D3QN network models for each of the four legs and adds a mask during action selection to form the D3QN_Mask algorithm, i.e., MD3QN. The foot placement planning algorithm for the tube sheet quadruped robot based on the D3QN algorithm is shown in Table 1.

[0120] Table 1. Flowchart of the foot planning algorithm based on MD3QN.

[0121]

[0122] Specific implementation method one:

[0124] This embodiment is a landing point planning system for a quadruped robot on a steam generator tube sheet. The system plans the landing points for the quadruped robot on the tube sheet. The quadruped robot on the tube sheet includes a base module and four leg modules. The four leg modules are the robot's four legs, which are set on the base module.

[0125] The system includes a status information storage module and a planning module;

[0126] State information storage module: Used to store the motion space and state space information of the tube sheet quadruped robot.

[0127] Action space:

[0128]

[0129] in, For each landing action, the corresponding landing point for each action. It is relative to the current foot position The actual landing position is k max It is the number of landing points that satisfy all possible cases of robot inverse kinematics, and it is also the dimension of the action space;

[0130] State space:

[0131] s C =[R, Pos T f]

[0132] state space s CIncludes the robot's position and orientation information R, and the endpoint Pos in the global motion direction. T (r T c T The criteria for determining whether a foot can actually be placed in the action space. Wherein, the position and attitude information R = [r B c B ,θ1,θ 21 θ 31 θ 22 θ 32 θ 23 θ 33 θ 24 θ 34 ], r B c B Here are the row and column coordinates of the robot's current position, θ1 is the rotation angle of the robot's base, and θ 21 θ 31 θ represents the two turning angles of the robot's left front leg. 22 θ 32 θ represents the two turning angles of the robot's left front leg. 23 θ 33 θ represents the two turning angles of the robot's left hind leg. 24 θ 34 For the two corners of the robot's right hind leg; r T c T The row and column coordinates of the endpoint in the global motion direction. f is a flag indicating whether a point in the robot's current action space can be placed. k =1 indicates that the point can be placed, f k =0 indicates that the point cannot be placed. The flag f(s) corresponding to state s constitutes the mask of state s.

[0133] Planning module: Reads the action space and state space from the state information storage module, and calls the D3QN model to plan the landing point. The reward function and action selection strategy used in the D3QN model's landing point planning process are as follows:

[0134] Reward function:

[0135] R C =-m0+R foot +m1R stable

[0136] R C It consists of three parts with different weights. The first part is the daily penalty, which is a penalty of m0 for each step, to ensure that the robot does not move too much or stop in place. The second part is the reward / penalty for moving forward / backward. The third part is about stable rewards / penalties.

[0137] R foot_k and R stable_k as follows:

[0138]

[0139]

[0140] Among them, l foot_k This represents the distance the foot travels along the direction of movement; a value greater than 0 indicates approaching the target, a value less than 0 indicates moving away from the target, and 0 indicates the foot is stationary in the direction of movement. foot_max This indicates the maximum distance the foot can travel along the direction of movement within the movement space; l c_to_sup_k This represents the distance between the centroid of the stable triangle formed by the current supporting feet and the base; l c_to_line_k l represents the distance from the centroid of the stable triangle formed by the current supporting feet to the line connecting the start and end points. stable_max This represents the farthest distance from the centroid of the stable triangle formed by the current supporting feet to the side of the triangle; m1, m2, ..., m8 are the parameters of the reward function;

[0141] Action selection strategy:

[0142]

[0143] Among them, random action from feasible A f (s) represents the list of feasible landing points. In the selection of random actions, 0≤ξ≤1 is a uniformly random number drawn in each time step, 0<∈<1 is a fixed probability in ∈-greedy, and Q(s,a) is the Q value corresponding to the selection of action a in state s.

[0144] In the process of foot placement planning using the D3QN model, the calculation formula for the target network in D3QN is as follows:

[0145]

[0146] Among them, y t Represents the target value at time t; γ is the discount factor, r t It is the reward function at time t, Q(s) t+1 a) is about state s t+1 And the state-action value function of the action, corresponding to The Q value of the target network, To evaluate the Q-value of the network; "next state is final state" means that the next state is the final state.

[0147] In the process of foot placement planning using the D3QN model, the formula for evaluating the value of network-computed state actions is as follows:

[0148]

[0149] in, It's about state s t The state value function, It's about state s t and action a t The dominance function, mean, represents the mean value, w, w V and w A These are the parameters for the corresponding network.

[0150] The evaluation network in D3QN uses a fully connected neural network model, which consists of an input layer, two hidden layers of size 64, and an output layer processed using the sigmoid activation function.

[0151] Example:

[0152] The evaporator tube sheet used in this invention can be simplified into a two-dimensional grid environment, and is divided into two types: one with square-shaped tube holes, such as... Figure 4 As shown in (a), one type of pipe has a triangular distribution of pores, such as... Figure 4 As shown in (b). Here, d1 = 27.43 and d2 = 25 are the tube hole spacings for the two types of tube sheets, and R1 = R2 = 8.865 are the tube hole radii for the two types of tube sheets. Obstacles were added to both the square and triangular tube sheets, and five tasks (initial position, initial attitude, and target position) were randomly set, resulting in ten different environments and tasks.

[0153] Given a straight line, the task is as follows: Figure 6 The robot's pose is represented by an 11-element tuple, where each element represents the base row coordinates, base column coordinates, base rotation angle, lf-axis 1 rotation angle, lf-axis 2 rotation angle, rf-axis 1 rotation angle, rf-axis 2 rotation angle, lb-axis 1 rotation angle, lb-axis 2 rotation angle, rb-axis 1 rotation angle, and rb-axis 2 rotation angle. The robot's initial pose for this task is [25, 60, 0, 30.49, -73.67, -30.49, 73.67, -29.17, 86.42, 29.17, -86.42], the initial position of the path is [25, 60], the ending position is [35, 60], and the path angle is 90°→.

[0154] We use the PyTorch network framework and Adam as the network optimizer. The MD3QN hyperparameters for training the network are as follows:

[0155] Table 2 MD3QN Hyperparameters

[0156]

[0157] n s m0, m2 and l foot_max Related, as shown below

[0158] Table 3 Algorithm parameters determined by environment and task

[0159]

[0160] Training begins. In the first round, the robot is reset to its initial state, and the neural network parameters of the evaluation network and the target network are reset. The initial landing leg is the lf leg. The action selection function is then entered, and all selectable actions are [[29,69],[29,70],[30,69],[30,70],[31,69],[31,70],[32,69],[32,70],[33,68],[33,69],[33,70],[34,67],[34,68],[34,69],[35,65],[35,66],[35,67],[35,68],[32,68]]. A random number is generated. The random number 0.6031 is less than the initial value ∈=1, so a random action is selected. The re-randomized action is [29,69]. This action is executed, interacting with the environment, resulting in a new state [26.27,60,0,-6.21,-82.69,-28.24,89.16,-30.22,69.49,30.23,-69.49]. The completion flag is "not completed," and the reward is -6.12. This set of <state, selected action, reward, new state, completion flag> is added to the experience pool, which now contains 1. The round continues with the gait sequence lf-rf-rb-lf until the experience pool contains more than 5000. At this point, the network parameters are updated by taking a batch of size 64 from the experience pool and evaluating the network parameters w. e Calculate the evaluation values ​​for these states, and obtain the target value based on the reward value derived from experience. Pass the target value and evaluation values ​​into the Adam optimizer for gradient descent calculation, updating the parameters of the evaluation network. Every 50 iterations, update the parameter values ​​of the target network and the evaluation network to match. Continue this process, starting the next round after each round ends (successfully reaching the endpoint or unable to move), resetting the robot's state to its initial state. Continue training until the maximum number of rounds is reached.

[0161] The training results for Square-1 (square tube sheet) and Triangle-1 (triangular tube sheet) are as follows: Figure 5 As shown, each curve represents the average and variance of a specified number of training iterations on an environment-task using 5 random seeds. Figure 5 (a) represents the average round reward and variance of the MD3QN and D3QN algorithms over 4000 training rounds in a square environment task. Figure 5 (b) represents the average number of rounds and the step variance of the MD3QN and D3QN algorithms over 4000 training rounds in a square environment task. Figure 5 (c) represents the average round reward and variance of the MD3QN and D3QN algorithms over 4000 training rounds in the triangular environment task. Figure 5 (d) represents the average number of epochs and the step variance of the MD3QN and D3QN algorithms over 4000 training epochs in the triangular environment task. The final training results for the 10 environments and tasks are shown in Table 4, where S represents a square tube sheet, T represents a triangular tube sheet, and the numbers indicate the task group. As can be seen from the table, under different environments and tasks, compared to the traditional D3QN algorithm, the D3QN_Mask algorithm achieves a higher total score, fewer steps, and better stability in the final foot placement planning result. The comparative experimental results demonstrate the effectiveness of the D3QN_Mask algorithm.

[0162] Table 4. Final training results for 10 environments and tasks

[0163]

[0164]

[0165] The above examples of the present invention are merely illustrative of the computational model and process of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is impossible to exhaustively list all possible implementations here. Any obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for planning the landing points of a quadruped robot on a steam generator tube sheet, comprising a base module and four leg modules; the four leg modules being the robot's four legs, which are mounted on the base module; characterized in that... The D3QN model was used to plan the landing points of the quadruped robot on the tube sheet. During the planning process, the motion space, state space, and reward function of the tube sheet quadruped robot are as follows: Action space: Where, a1, a2, ..., For each landing action, the corresponding landing point for each action. It is relative to the current foot position The actual landing position is k max k max It is the number of landing points that satisfy all possible cases of robot inverse kinematics, and it is also the dimension of the action space; State space: s C = [R, Pos T , f] state space s C Includes the robot's position and orientation information R, and the endpoint Pos in the global motion direction. T (r T c T The criteria for determining whether a foot can actually be placed in the action space. Where r T c T The row and column coordinates of the endpoint in the global motion direction. f is a flag indicating whether a point in the robot's current action space can be placed. k =1 indicates that the point can be placed, f k =0 indicates that the point cannot be placed. The flag f(s) corresponding to state s constitutes the mask of state s. Reward function: R C =-m0+R foot +m1R stable R C It consists of three parts with different weights. The first part is the daily penalty, which is a penalty of m0 for each step, to ensure that the robot does not move too much or stop in place. The second part is the reward / penalty for moving forward / backward. The third part is about stable rewards / penalties. R foot_k and R stable_k as follows: Among them, l foot_k This represents the distance the foot travels along the direction of movement; a value greater than 0 indicates approaching the target, a value less than 0 indicates moving away from the target, and 0 indicates the foot is stationary in the direction of movement. foot_max This indicates the maximum distance the foot can travel along the direction of movement within the movement space; l c_to_sup_k This represents the distance between the centroid of the stable triangle formed by the current supporting feet and the base; l c_to_line_k l represents the distance from the centroid of the stable triangle formed by the current supporting feet to the line connecting the start and end points. stable_max This represents the farthest distance from the centroid of the stable triangle formed by the current supporting feet to the side of the triangle; m1, m2, ..., m8 are the parameters of the reward function; Based on the aforementioned action space, state space, and reward function, the action selection strategy for foot placement planning using the D3QN model is as follows: Among them, random action from feasible A f (s) represents the list of feasible landing points. In the selection of random actions, 0≤ξ≤1 is a uniformly random number drawn in each time step, 0<∈<1 is a fixed probability in ∈-greedy, and Q(s,a) is the Q value corresponding to the selection of action a in state s.

2. The method for planning the landing points of a quadruped robot on a steam generator tube sheet according to claim 1, characterized in that, state space s C Position and attitude information R = [r B c B ,θ1,θ 21 θ 31 θ 22 θ 32 θ 23 θ 33 θ 24 θ 34 ], where r B c B Here are the row and column coordinates of the robot's current position, θ1 is the rotation angle of the robot's base, and θ 21 θ 31 θ represents the two turning angles of the robot's left front leg. 22 θ 32 θ represents the two turning angles of the robot's left front leg. 23 θ 33 θ represents the two turning angles of the robot's left hind leg. 24 θ 34 These are the two corners of the robot's right hind leg.

3. The method for planning the landing points of a quadruped robot on a steam generator tube sheet according to claim 2, characterized in that, In the process of foot placement planning using the D3QN model, the calculation formula for the target network in D3QN is as follows: Among them, y t Represents the target value at time t; γ is the discount factor, r t It is the reward function at time t, Q(s) t+1 a) is about state s t+1 And the state-action value function of the action, corresponding to The Q value of the target network, To evaluate the Q-value of the network; the next state is the final state.

4. The method for planning the landing points of a quadruped robot on a steam generator tube sheet according to claim 3, characterized in that, In the process of foot placement planning using the D3QN model, the formula for evaluating the value of network-computed state actions is as follows: Among them, V wV (s t ) is about state s t The state value function, It's about state s t and action a t The dominance function, mean, represents the mean value, w, w V and w A These are the parameters for the corresponding network.

5. A method for planning the landing points of a quadruped robot on a steam generator tube sheet according to any one of claims 1 to 4, characterized in that, The evaluation network in D3QN uses a fully connected neural network model, which consists of an input layer, two hidden layers of size 64, and an output layer processed using the sigmoid activation function.

6. A landing point planning system for a quadruped robot on a steam generator tube sheet, the system planning the landing points of the quadruped robot on the tube sheet, the quadruped robot comprising a base module and four leg modules; the four leg modules being the robot's four legs, which are mounted on the base module; characterized in that, The system includes a status information storage module and a planning module; State information storage module: Used to store the motion space and state space information of the tube sheet quadruped robot. Action space: Where, a1, a2, ..., For each landing action, the corresponding landing point for each action. It is relative to the current foot position The actual landing position is kmax, kmax is the number of landing points that satisfy all possible cases of robot inverse kinematics, and it is also the dimension of the action space; State space: s C =[R,Pos T ,f] state space s C Includes the robot's position and orientation information R, and the endpoint Pos in the global motion direction. T (r T c T The criteria for determining whether a foot can actually be placed in the action space. Where r T c T The row and column coordinates of the endpoint in the global motion direction. f is a flag indicating whether a point in the robot's current action space can be placed. k =1 indicates that the point can be placed, f k =0 indicates that the point cannot be placed. The flag f(s) corresponding to state s constitutes the mask of state s. Planning module: Reads the action space and state space from the state information storage module, and calls the D3QN model to plan the landing point. The reward function and action selection strategy used in the D3QN model's landing point planning process are as follows: Reward function: R C =-m o +R foot +m1R stable R C It consists of three parts with different weights. The first part is the daily penalty, which is a penalty of m0 for each step, to ensure that the robot does not move too much or stop in place. The second part is the reward / penalty for moving forward / backward. The third part is about stable rewards / penalties. R foot_k and R stable_k as follows: Among them, l foot_k This represents the distance the foot travels along the direction of movement; a value greater than 0 indicates approaching the target, a value less than 0 indicates moving away from the target, and 0 indicates the foot is stationary in the direction of movement. foot_max This indicates the maximum distance the foot can travel along the direction of movement within the movement space; l c_to_sup_k This represents the distance between the centroid of the stable triangle formed by the current supporting feet and the base; l c_to_line_k l represents the distance from the centroid of the stable triangle formed by the current supporting feet to the line connecting the start and end points. stable_max This represents the farthest distance from the centroid of the stable triangle formed by the current supporting feet to the side of the triangle; m1, m2, ..., m8 are the parameters of the reward function; Action selection strategy: Among them, random action from feasible A f (s) represents the list of feasible landing points. In the selection of random actions, 0≤ξ≤1 is a uniformly random number drawn in each time step, 0<∈<1 is a fixed probability in ∈-greedy, and Q(s,a) is the Q value corresponding to the selection of action a in state s.

7. A landing point planning system for a quadruped robot on a steam generator tube sheet according to claim 6, characterized in that, state space s C Position and attitude information R = [r B c B ,θ1,θ 21 θ 31 θ 22 θ 32 θ 23 θ 33 θ 24 θ 34 ], where r B c B Here are the row and column coordinates of the robot's current position, θ1 is the rotation angle of the robot's base, and θ 21 θ 31 θ represents the two turning angles of the robot's left front leg. 22 θ 32 θ represents the two turning angles of the robot's left front leg. 23 θ 33 θ represents the two turning angles of the robot's left hind leg. 24 θ 34 These are the two corners of the robot's right hind leg.

8. A landing point planning system for a quadruped robot on a steam generator tube sheet according to claim 7, characterized in that, In the process of foot placement planning using the D3QN model, the calculation formula for the target network in D3QN is as follows: Among them, y t Represents the target value at time t; γ is the discount factor, r t It is the reward function at time t, Q(s) t+1 a) is about state s t+1 And the state-action value function of the action, corresponding to The Q value of the target network, To evaluate the Q-value of the network; "next state is final state" means that the next state is the final state.

9. A landing point planning system for a quadruped robot on a steam generator tube sheet according to claim 8, characterized in that, In the process of foot placement planning using the D3QN model, the formula for evaluating the value of network-computed state actions is as follows: in, It's about state s t The state value function, It's about state s t and action a t The dominance function, mean, represents the mean value, w, w V and w A These are the parameters for the corresponding network.

10. A landing point planning system for a quadruped robot on a steam generator tube sheet according to any one of claims 6 to 9, characterized in that, The evaluation network in D3QN uses a fully connected neural network model, which consists of an input layer, two hidden layers of size 64, and an output layer processed using the sigmoid activation function.

Images