Contact type machining robot motion planning method based on deep reinforcement learning

Abstract

The invention relates to a contact type machining robot motion planning method based on deep reinforcement learning, and the method comprises the steps: firstly, building position-level forward and inverse kinematics models between the tail end state vector of a six-degree-of-freedom mechanical arm and the motion vectors of a base and a joint based on DH parameters and a homogeneous transformation matrix; secondly, constraints such as a machining task, a feeding speed and a feeding angle are comprehensively considered, and a multi-stage reward function is designed to plan operation parameters; secondly, designing a network structure based on a double-delay depth deterministic strategy gradient, and performing three-level optimization on the algorithm; and finally, designing a planning process according to a network structure, and applying the planning process to a planning task of an intelligent contact type machining mechanical arm system by taking grinding and polishing machining as an example. According to the method, real-time optimization and control of various machining parameters of the contact type machining process can be realized through autonomous learning, and the method can be used for intelligent safety planning under the dynamic coupling condition of the contact type machining process.

Description

Technical Field

[0001] The present invention belongs to the field of motion planning of intelligent contact processing robots, and specifically relates to a motion planning method of a contact processing robot based on deep reinforcement learning. Background Art

[0002] There are multiple cross-coupling constraints in the operation of multi-joint industrial robots. At the same time, there are unexpected rigid contact effects in contact processing tasks such as grinding, polishing, and hole making. Improper processing parameter settings can easily cause self-excited tangential chatter with temporal and spatial variations, resulting in poor workpiece processing quality and low processing efficiency. Therefore, it is urgent to develop a real-time safety planning method and parameter optimization technology to track and plan the entire processing process.

[0003] In the motion planning of contact processing robots, the traditional method usually plans the tool path points. There are three main methods: equal chord length error method, equal chord height error method, and equal parameter method. However, the traditional methods only use the curvature change and normal direction of the surface as the standard, aiming to optimize the path and improve the processing efficiency from the perspective of the algorithm, and do not consider the continuity and smoothness of the robot path in the actual processing process. With the development of machine learning, a Chinese patent application (application number CN202410322517.6) proposed a method for learning and planning the marked two-dimensional data after converting the workpiece data into a two-dimensional model, but this method ignores the problem of trajectory continuity during the processing process and cannot be planned in real time. For welding robots, a Chinese patent application (application number CN202410667799.3) proposed a data processing planning method based on deep learning, but this method requires multiple calls to the model, and the planning results are affected by the call time, resulting in large differences. For the tool planning problem, a Chinese patent application (application number CN202410291785.6) proposed a method of compensating for machining errors using a deep reinforcement learning algorithm. Although this method effectively reduces machining errors, its pre-planning of the tool relies on the machining model and is not accurate enough. Summary of the invention

[0004] Aiming at the problem that the robot working environment has different unstructured dynamic characteristics in different contact processing tasks, and the working state is affected by factors such as process parameters and robot parameters, the present invention proposes a contact processing robot motion planning method based on deep reinforcement learning. It does not require precise modeling of the processing process, and can achieve real-time optimization and control of multiple processing parameters of the contact processing process through autonomous learning, which can be used for intelligent safety planning under the dynamic coupling of the contact processing robot.

[0005] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0006] A motion planning method for a contact processing robot based on deep reinforcement learning, the method comprising the following steps:

[0007] Firstly, based on DH parameters and homogeneous transformation matrices, the position-level forward and inverse kinematic models between the state variables of the end state of the six-degree-of-freedom robot and the base and joint motion variables were established; secondly, considering the constraints such as processing tasks, feed speed and feed angle, a multi-level reward function based on multiple constraints was designed to plan the operation parameters; then, based on the double-delay deep deterministic policy gradient, the network structure was designed, and a triple-improved double-delay deep deterministic policy gradient network was established; finally, the triple-improved double-delay deep deterministic policy gradient network was trained and loaded to plan the motion trajectory of the robot and optimize the intelligent safety parameters of contact processing. Taking grinding and polishing as an example, it was applied to the planning task of the intelligent contact processing robot arm system.

[0008] The first step is specifically implemented as follows:

[0009] Establish the forward kinematics model of the robot arm, and transform the i-1th coordinate system of the robot arm link to the ith coordinate system according to the modified DH parameter representation method The general formula is:

[0010] ,

[0011] In DH parameter representation, represents the angle between the joint of the ith coordinate system and the two adjacent equivalent straight rods, represents the length of the equivalent straight rod of the connecting rod in the i-th coordinate system, represents the angle between two adjacent joint axes of the link in the i-th coordinate system, represents the distance between two adjacent equivalent straight rods connected to the joint in the i-th coordinate system;

[0012] Homogeneous change matrix of tool coordinate system at the end of the robot arm relative to the inertial coordinate system for:

[0013] ,

[0014] In the formula, is the homogeneous transformation matrix from the inertial coordinate system to the base mass center coordinate system, is the homogeneous transformation matrix from the base mass center coordinate system to the robot arm installation coordinate system, is the homogeneous transformation matrix from the end joint coordinate system to the robot arm installation coordinate system, is the homogeneous transformation matrix from the end joint coordinate system to the tool coordinate system at the end of the robot arm;

[0015] Establish an inverse kinematics model for the manipulator, classify the joints of the manipulator with SRS configuration (three ball joints and three revolute joints) into non-coplanar joint groups and coplanar parallel axis joint groups, solve the non-coplanar joint group by left-multiplying the inverse matrix and making the corresponding elements equal, then bring the solution of the non-coplanar joint group into the solution, and solve the coplanar parallel axis joint group according to the properties of trigonometric functions;

[0016] The following assumptions are made for the grinding and polishing process:

[0017] (1) The contact area is small compared to the disc and workpiece geometry;

[0018] (2) The workpiece surface is smooth and the surface equation is continuous to the second order;

[0019] (3) The polishing disc is much softer than the workpiece, so there is no deformation of the workpiece;

[0020] (4) The deformation of the disk is caused only by the normal contact force, and the tangential contact force is not considered;

[0021] (5) Compared with the workpiece curvature radius, the material removal depth during polishing is negligible.

[0022] The machining contact surface model is established, and the geometric structure of the machining tool flexible grinding disc is equivalent to a disc. The disc with a radius of R is assumed to contact the workpiece at the initial point O. A standard orthogonal coordinate system O-XYZ is established with O as the origin. The Z axis is the direction of the surface normal, the X axis is along the tool feed direction, Od is the center of the disc bottom surface, and the Y axis is given by the right-hand rule. The contact depth of the disc Defined as the displacement of the disk along the surface normal, the disk tilt angle Defined as the angle between the bottom surface of the disk and the tangent plane of the workpiece, the flexible disk is tilted at an angle Press into the workpiece to the contact depth After that, the bottom of the disk produces a local deformation at the initial point O, and the plane expressed by the bottom of the disk Written in the following form:

[0023] ,

[0024] x and y represent the two-dimensional coordinates in the plane.

[0025] According to the theory of classical differential geometry, the smooth workpiece contour near the contact point O is approximated as a quadratic function:

[0026] ,

[0027] In the formula represents the principal curvature of the contact surface, represents the secondary curvature of the contact surface, represents the angle between the feeding direction of the flexible grinding disc and the principal curvature of the contact surface, and T represents the transpose of the matrix.

[0028] The second step is specifically implemented as follows:

[0029] (1) Constructing a reward function based on processing task constraints :

[0030] ,

[0031] In the formula, d represents the distance between the end effector and the corresponding processing point, k1 and k2 are the first and second proportional coefficients;

[0032] (2) Constructing a reward function based on feed rate constraints ;

[0033] The size and direction of the feed speed are important factors affecting the grinding and polishing effect. When the speed of the flexible grinding disc at the end is too high, a collision may occur, causing damage to the workpiece and the driving robot arm. In order to meet this constraint, the speed constraint is ignored when the end is far from the target position. When the distance between the end and the target position is within a safe range, a penalty function of speed and distance is established:

[0034] ,

[0035] Where v e represents the feed speed of the flexible grinding disc, k3 represents the proportional coefficient of the speed constraint affected by the distance, and k4 represents the proportional coefficient affecting the speed constraint reward value.

[0036] (3) Constructing a reward function based on feed angle constraints ;

[0037] The feed angle constraint is essentially a constraint on the end-arm posture. In the contact surface modeling, the angle between the grinding wheel feed direction and the principal curvature of the contact surface is defined. , Too large a value will generate a large tangential force and cause damage to the tool. The designed reward function is as follows:

[0038] ,

[0039] Where k5 represents the proportional coefficient of the degree to which the angle constraint reward value is affected by the distance, and k6 represents the proportional coefficient that affects the angle constraint reward value.

[0040] Taking the triple constraints into consideration, the comprehensive reward function is proposed as follows:

[0041] ,

[0042] The third step is specifically implemented as follows:

[0043] A dual actor-critic framework is used to fit the value function and the policy function. An experience replay buffer is introduced to implement batch checking during the training process. And a triple improvement is made to the network to improve the efficiency of the algorithm.

[0044] First, simplify the two Q networks. In the Actor-Critic network, the update strategy is:

[0045] ,

[0046] Where r represents the cumulative reward value, γ is the discount factor, is the policy function.

[0047] In the actual calculation process, when the Actor-Critic network updates slowly, the two networks are similar and do not have independent conditions to make independent estimates. Therefore, a biased estimation function is given.

[0048] ,

[0049] , Represent the biased estimation functions of the two Actor-Critic networks respectively; , Represent the strategy functions of the two Actor-Critic networks respectively;

[0050] Select the smaller value between the two as the final estimate .

[0051] The second is to propose a delayed update strategy. During the learning process, the target network is used as a function approximator. The target network provides a stable target for the learning process as a control, which accelerates the convergence speed of the network and greatly improves the stability. In order to minimize error propagation, the Actor network is updated less frequently than the Critic network. The lower the policy update frequency, the smaller the update variance of the value function, thereby obtaining a higher quality policy. By sufficiently delaying the policy update, the possibility of repeated updates for unchanged critics is limited.

[0052] The third point is to smooth the target strategy so that the action has a normalized range. A regularization strategy is introduced to map similar values ​​to similar action spaces, and the value is regularized to a region in the action space. This is done by adding a small amount of normally distributed noise to the action and averaging the small batch experience pool.

[0053] The fourth step is specifically implemented as follows:

[0054] The process of designing the algorithm is as follows:

[0055] 1) First initialize the Critic network With Actor Network , setting the state vector and assigning a random initial value, wherein the state vector includes the joint angle and angular velocity of the robot arm, the velocity and posture of the end effector of the robot arm, and the distance between the end effector and the workpiece;

[0056] 2) Initialize the target critic network With Actor Network , initialize the experience buffer pool R;

[0057] 3) Randomly give a target planning position and obtain the current state s of the robot arm;

[0058] 4) Select the current action after adding random noise according to the current state s , Nt is random noise;

[0059] 5) Execute the current action And obtain the reward value r and the new state s' according to the designed reward function, and Stored in the experience buffer;

[0060] 6) Randomly sample N groups of samples in the experience buffer to form a minimum batch set;

[0061] 7) According to the optimization method described in step 3, normalize the actions and simplify the dual Q network;

[0062] 8) Update the Critic network according to the double-delay update strategy and loss function described in step 3;

[0063] 9) Update the Actor network based on the gradient update function;

[0064] 10) Update the target network;

[0065] 11) End.

[0066] The beneficial effects of the present invention are:

[0067] Through the deep reinforcement learning method, the whole process of contact processing motion is intelligently planned to ensure the safety and reliability of processing parameters and avoid self-excited vibration between the robot arm and the workpiece due to improper parameters. The present invention is not limited to grinding and polishing planning, but is also applicable to other types of contact processing such as welding and spraying. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 A flowchart of a motion planning method for a contact processing robot based on deep reinforcement learning according to the present invention;

[0069] Figure 2 The modeling structure diagram of the DH section card mechanical arm system of the present invention;

[0070] Figure 3 Processing a three-dimensional model of the contact surface for the present invention;

[0071] Figure 4 This is a structural diagram of the dual-delay deep deterministic policy gradient network of the present invention. DETAILED DESCRIPTION

[0072] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0073] like Figure 1 As shown, the present invention provides a contact processing robot motion planning method based on deep reinforcement learning, comprising:

[0074] In the first step, a general dynamics model of contact machining is obtained based on the position-level forward and inverse robot coupling kinematics model of the manipulator system and the machining contact surface model;

[0075] The second step is to construct a multi-level reward function based on multiple constraints and plan the operation parameters;

[0076] The third step is to design an algorithm structure based on the dual Actor-Critic framework based on the general dynamics model of contact processing, and establish a triple improved dual-delay deep deterministic policy gradient network;

[0077] The fourth step is to train and load the triple improved dual-delay deep deterministic policy gradient network, plan the motion trajectory of the robot, and optimize the intelligent safety parameters of contact processing.

[0078] Example

[0079] Taking the grinding and polishing robot as an example, the first step is specifically implemented as follows:

[0080] The grinding and polishing robot system studied in this invention selects the joint card robot arm, and the definition of the DH coordinate system is shown in Figure 2 , each joint coordinate system is defined as , the corresponding DH parameters are shown in Table 1, where Indicates the length of the i-th arm, the installation coordinate system of the robot arm Coordinate system with the center of mass of the base Pointing the same, the installation origin is at the center of mass of the base:

[0081] Table 1

[0082]

[0083] Solve the forward kinematics model of the robotic arm:

[0084] Connecting rod transformation based on modified DH parameter representation The general formula is:

[0085] ,

[0086] In the DH parameter representation, θi represents the angle between the two adjacent equivalent straight rods adjacent to joint i, ai represents the length of the equivalent straight rod of link i, αi represents the angle between the two adjacent joint axes of link i, and di represents the distance between the two adjacent equivalent straight rods of the necklace of joint i.

[0087] Homogeneous change matrix of tool coordinate system at the end of the robot arm relative to the inertial coordinate system for:

[0088] ,

[0089] In the formula, is the homogeneous transformation matrix from the inertial coordinate system to the base mass center coordinate system, is the homogeneous transformation matrix from the base mass center coordinate system to the robot arm installation coordinate system, is the homogeneous transformation matrix from the end joint coordinate system to the robot arm installation coordinate system, is the homogeneous transformation matrix from the end joint coordinate system to the tool coordinate system at the end of the robot arm.

[0090] The inverse kinematics of the robot arm is calculated, and the joints of the robot arm with SRS configuration (three ball joints and three revolute joints) are classified into non-coplanar joint groups and coplanar parallel axis joint groups for solution.

[0091] like Figure 3 As shown, the geometric modeling process is given by taking grinding and polishing as an example.

[0092] The following assumptions are made for the grinding and polishing process:

[0093] (1) The contact area is small compared to the disc and workpiece geometry;

[0094] (2) The workpiece surface is smooth and the surface equation is continuous to the second order;

[0095] (3) The polishing disc is much softer than the workpiece, so there is no deformation of the workpiece;

[0096] (4) The deformation of the disk is caused only by the normal contact force, and the tangential contact force is not considered;

[0097] (5) Compared with the workpiece curvature radius, the material removal depth during polishing is negligible.

[0098] Assume that a disk with a radius of R = 200 mm and a height of H = 50 mm contacts the workpiece at the initial point O. A standard orthogonal coordinate system O-XYZ is established with O as the origin, the Z axis is the direction of the surface normal, the X axis is along the tool feed direction, and Od is the center of the disk bottom surface. The Y axis is given by the right-hand rule. The disk contact depth Defined as the displacement of the disk along the surface normal, the disk tilt angle Defined as the angle between the bottom surface of the disk and the tangent plane of the workpiece, the flexible disk is tilted at an angle Press into the workpiece to the contact depth After that, the bottom of the disk produces a local deformation at the initial point O, and the plane expressed by the bottom of the disk Written in the following form:

[0099] ,

[0100] x and y represent the two-dimensional coordinates in the plane.

[0101] According to the theory of classical differential geometry, the smooth workpiece contour near the contact point O is approximated as a quadratic function:

[0102] ,

[0103] In the formula, represents the principal curvature of the contact surface, represents the secondary curvature of the contact surface, represents the angle between the feeding direction of the flexible grinding disc and the principal curvature of the contact surface, and T represents the transpose of the matrix.

[0104] The second step is specifically implemented as follows:

[0105] (1) Task constraints

[0106] The distance between the end effector and the corresponding capture point (denoted as d) is the key to the design of the reward function. In order to make the reward increase as the relative distance decreases, the reward function is taken as A positive proportional function in the form of , when the end effector reaches a small area near the target, it is expected to give the manipulator an increasing reward for accelerating to the target point. To this end, a logarithmic function is introduced:

[0107] ,

[0108] In the formula, The term avoids the singularity of the logarithmic function and limits the maximum value of the reward. k1=2, as d gradually decreases, the reward first increases at a constant rate, and the end is evenly planned to a closer position. k2=0.5, when d is reduced to within 1 meter, the reward value increases significantly, further improving the accuracy of planning and achieving the design purpose.

[0109] (2) Feed speed constraint

[0110] The size and direction of the feed speed are important factors affecting the grinding and polishing effect. When the speed of the flexible grinding disc at the end is too high, a collision may occur, causing damage to the workpiece and the driving robot arm. When the end is far from the target position, the speed constraint will be ignored. When the distance between the end and the target position is within a safe range, a penalty function of speed and distance is established:

[0111] ,

[0112] The penalty function designed by the analysis is: e represents the feed speed of the grinding disc, d represents the distance between the grinding disc and the workpiece. k3=1 determines the intensity of the penalty, that is, the degree to which the speed constraint is affected by the distance, and k4=0.1 determines the maximum value of the penalty.

[0113] (3) Feed angle constraint

[0114] The feed angle constraint is essentially a constraint on the end-arm posture. In the contact surface modeling, the angle between the grinding wheel feed direction and the principal curvature of the contact surface is defined. , Too large a tangential force will cause damage to the tool. Therefore, it is necessary to plan the posture of the end effector of the robot arm during processing to control the feed angle so that Within a safe range, the designed reward function is as follows:

[0115] ,

[0116] Analyze the designed reward function. When the distance is large, no constraints are imposed on the robot arm posture. k5=1 determines the extent to which the reward value is affected by the distance, that is, the size of the safety distance. k6=0.2 affects the size of the reward value.

[0117] Taking the triple constraints into consideration, the comprehensive reward function is proposed as follows:

[0118] ,

[0119] The third step is specifically implemented as follows:

[0120] The designed network structure is as follows Figure 4 As shown in the figure, a dual Actor-Critic framework is used to build the target network and the online network. An experience replay buffer is introduced to achieve batch inspection of the training process. And the network is improved in three ways to improve the efficiency of the algorithm.

[0121] First, simplify the two Q networks. In the Actor-Critic network, the update strategy is:

[0122] ,

[0123] Where r represents the cumulative reward value, γ is the discount factor, is the policy function.

[0124] In the actual calculation process, when the Actor-Critic network updates slowly, the two networks are similar and do not have independent conditions to make independent estimates. Therefore, a biased estimation function is given.

[0125] ,

[0126] Select the smaller value between the two as the final estimate .

[0127] The second is to propose a delayed update strategy. During the learning process, the target network is used as a function approximator. The target network provides a stable target for the learning process as a control, which accelerates the convergence speed of the network and greatly improves the stability. In order to minimize error propagation, the policy network is designed to be updated at a lower frequency than the value network. The lower the policy update frequency, the smaller the update variance of the value function, thereby obtaining a higher quality policy. By sufficiently delaying the policy update, the possibility of repeated updates for unchanged critics is limited.

[0128] The third point is to smooth the target strategy so that the action has a normalized range. A regularization strategy is introduced to map similar values ​​to similar action spaces, and the value is regularized to a region in the action space. This is done by adding a small amount of normally distributed noise to the action and averaging the small batch experience pool.

[0129] The fourth step is specifically implemented as follows:

[0130] 1) If Figure 4 As shown, first initialize the Critic network of the online network With Actor Network , set the state vector as: the robot arm joint angle and angular velocity, the speed and posture of the robot arm end effector, the distance between the end effector and the workpiece, etc., and assign random initial values ​​to the state;

[0131] 2) Initialize the Critic network of the target network With Actor Network , initialize the experience buffer R and set the upper limit of R to 10000;

[0132] 3) Randomly give a target planning position and obtain the current state s of the robot arm;

[0133] 4) Select the current action after adding detection noise according to the current state s , N t For random noise, set it to Gaussian noise;

[0134] 5) Execute the action and obtain the reward value r and the new state s', and Stored in the experience pool;

[0135] 6) Randomly sample N=64 groups of samples in the experience pool to form a minimum batch set;

[0136] 7) According to the optimization method described in step 3, normalize the actions and simplify the dual Q network;

[0137] 8) According to the delayed update strategy described in step 3, update the Critic network according to the loss function;

[0138] 9) Update the Actor network based on the policy gradient;

[0139] 10) Update the target network;

[0140] 11) End.

[0141] Based on the triple improvement, the learning parameters are selected as follows: the Actor network has two hidden layers, each layer has 200 neurons, and the learning rate is 1.0×10-4; the Critic network has two hidden layers, each layer has 200 neurons, and the learning rate is 1.0×10-4; the total number of learning rounds is 5000 rounds, and the maximum number of steps in a single round is 128 steps; after the network converges, the network is loaded into the joint card robot arm, and the planning instructions are issued. During the planning process, the robot arm will update the information in the state vector in real time after each step of action, and call the Actor-critic network based on the current state value to obtain the next step of instructions. Thereby achieving a safe and stable approach to the workpiece, and the feed speed and feed angle are the target values. The technology of the present invention is not only limited to grinding and polishing planning, but is also applicable to other types of contact processing processes such as welding and spraying.

[0142] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A motion planning method for a contact processing robot based on deep reinforcement learning, characterized in that: The following steps are involved: In the first step, a general dynamics model of contact machining is obtained based on the position-level forward and inverse robot coupling kinematics model of the manipulator system and the machining contact surface model; The second step is to construct a multi-level reward function based on multiple constraints and plan the operation parameters; The third step is to design an algorithm structure based on the dual Actor-Critic framework based on the general dynamics model of contact processing, and establish a triple improved dual-delay deep deterministic policy gradient network; The fourth step is to train and load the triple improved dual-delay deep deterministic policy gradient network, plan the motion trajectory of the robot, and optimize the intelligent safety parameters of contact processing.

2. According to claim 1, a contact processing robot motion planning method based on deep reinforcement learning is characterized in that: The first step is specifically implemented as follows: Establish the forward kinematics model of the robot arm, and transform the i-1th coordinate system of the robot arm link to the ith coordinate system according to the modified DH parameter representation method The general formula is: ， In DH parameter representation, represents the angle between the joint of the ith coordinate system and the two adjacent equivalent straight rods, represents the length of the equivalent straight rod of the connecting rod in the i-th coordinate system, represents the angle between two adjacent joint axes of the link in the i-th coordinate system, represents the distance between two adjacent equivalent straight rods connected to the joint in the i-th coordinate system; Homogeneous change matrix of tool coordinate system at the end of the robot arm relative to the inertial coordinate system for: ， In the formula, is the homogeneous transformation matrix from the inertial coordinate system to the base mass center coordinate system, is the homogeneous transformation matrix from the base mass center coordinate system to the robot arm installation coordinate system, is the homogeneous transformation matrix from the end joint coordinate system to the robot arm installation coordinate system, is the homogeneous transformation matrix from the end joint coordinate system to the tool coordinate system at the end of the robot arm; Establish an inverse kinematics model of the manipulator, classify the manipulator joints of the SRS configuration into a non-coplanar joint group and a coplanar parallel axis joint group for solution, the SRS configuration includes three ball joints and three revolute joints, solve the non-coplanar joint group by left-multiplying the inverse matrix and making the corresponding elements equal, then bring the solution of the non-coplanar joint group into the solution, and solve the coplanar parallel axis joint group according to the properties of trigonometric functions; Establish a machining contact surface model, and equate the geometric structure of the machining tool flexible grinding disc to a disc. Assume that the disc with a radius of R contacts the workpiece at the initial point O. Establish a standard orthogonal coordinate system O-XYZ with O as the origin. The Z axis is the direction of the surface normal, the X axis is along the tool feed direction, Od is the center of the disc bottom surface, and the Y axis is given by the right-hand rule. The contact depth of the disc Defined as the displacement of the disk along the surface normal, the disk tilt angle Defined as the angle between the bottom surface of the disk and the tangent plane of the workpiece, the flexible disk is tilted at an angle Press into the workpiece to the contact depth After that, the bottom of the disk produces a local deformation at the initial point O, and the plane expressed by the bottom of the disk Written in the following form: ， x, y represent the two-dimensional coordinates in the plane; According to the theory of classical differential geometry, the smooth workpiece contour within the preset range of the initial point O is approximated as a quadratic function : ， In the formula, represents the principal curvature of the contact surface, represents the secondary curvature of the contact surface, represents the angle between the feeding direction of the flexible grinding disc and the principal curvature of the contact surface, and T represents the transpose of the matrix; Based on the established manipulator position-level forward and inverse robot kinematics models and machining contact surface model, a general contact machining model of the contact machining robot is obtained.

3. The motion planning method of a contact processing robot based on deep reinforcement learning according to claim 1, characterized in that: The second step is specifically implemented as follows: Constructing a reward function based on processing task constraints : ， In the formula, d represents the distance between the end effector and the corresponding processing point, k1 and k2 are the first and second proportional coefficients; Constructing a reward function based on the feedrate constraint : ， In the formula, v e represents the feed speed of the flexible grinding disc, k3 represents the proportional coefficient of the speed constraint affected by the distance, and k4 represents the proportional coefficient affecting the speed constraint reward value; Constructing a reward function based on the feed angle constraint : ， In the formula, represents the angle between the grinding disc feed direction and the principal curvature of the contact surface, k5 represents the proportional coefficient of the degree to which the angle constraint reward value is affected by the distance, and k6 represents the proportional coefficient that affects the angle constraint reward value; Given a comprehensive reward function for: 。 4. The motion planning method of a contact processing robot based on deep reinforcement learning according to claim 1, characterized in that: The third step is specifically implemented as follows: The first improvement includes: both Q networks are Actor-Critic networks: ， , Represent the biased estimation functions of the two Actor-Critic networks respectively; , Represent the strategy functions of the two Actor-Critic networks respectively; Select , The smaller value between the two is used as the estimated function estimator ; represents the discount factor; The second improvement includes: during the learning process, the target network is used as a function approximator to update the network, and the Actor network is updated less frequently than the Critic network; The third improvement includes: introducing a regularization strategy to regularize the value to a region in the action space.

5. The motion planning method of a contact processing robot based on deep reinforcement learning according to claim 1, characterized in that: The fourth step is specifically implemented as follows: 1) Initialize the Critic network of the online network With Actor Network , setting the state vector and assigning a random initial value, wherein the state vector includes the joint angle and angular velocity of the robot arm, the velocity and posture of the end effector of the robot arm, and the distance between the end effector and the workpiece; 2) Initialize the Critic network of the target network With Actor Network , initialize the experience buffer; 3) Randomly give the target planning position and obtain the current state s of the robot arm; 4) Select the current action after adding random noise according to the current state s , N t is the random noise added; 5) Execute the current action And according to the designed reward function Get the reward value and the new state s', and Stored in the experience buffer; 6) Randomly sample N groups of samples in the experience buffer to form a minimum batch set; 7) According to the optimization method described in step 3, normalize the actions and simplify the dual Q network; 8) Update the Critic network according to the delayed update strategy and loss function described in step 3; 9) Update the Actor network based on the policy gradient function; 10) Update the target network; 11) End.

Images