A method for adaptive compliant force control of a robot arm based on reinforcement learning

Abstract

This invention discloses an adaptive compliant force control method for robotic arms based on reinforcement learning, belonging to the fields of robotics and automation. This method is based on an integrated framework of "reinforcement learning-adaptive compliant force control" for a robotic arm-dexterous hand-six-dimensional force / torque sensor system. It aims to completely eliminate existing problems such as the need for manual parameter tuning during model changes, difficulties in contact dynamics modeling, model parameter estimation, and the frequent need to prove the stability of adaptive laws through intelligent control strategies. It achieves self-learning and self-adaptation of upper-level task planning and lower-level compliant force / position control. After deployment, no repeated manual parameter tuning is required for different working conditions. The same control strategy maintains a high assembly success rate and low peak contact force for workpieces with small gaps, easy deformation, and multiple materials, eliminating bottlenecks in manual calibration, environmental modeling, and online parameter estimation, and enabling rapid "zero-downtime" switching in flexible production lines.

Description

Technical Field

[0001] This invention belongs to the fields of robotics and automation, and particularly relates to an adaptive compliant force control method for a robotic arm based on reinforcement learning. Background Technology

[0002] As collaborative and humanoid robots enter mass production, the ability to "work alongside humans" has become a rigid requirement, placing higher demands on end-effector force control capabilities. Traditional industrial robotic arms focus on positional accuracy and cycle time, with control architectures mostly consisting of single-loop position servos. During contact operations, if the trajectory does not match the workpiece, overshoot, jamming, or even damage can easily occur. Collaborative and humanoid arms, on the other hand, need to complete dual-constraint tasks such as insertion / removal, screw tightening, and human-robot collaboration, requiring "millimeter-level position + supermassive force." This has spurred research into force-position hybrid control technology.

[0003] To achieve compliance, the industry has developed two main approaches: "direct force control" and "indirect force control." Admittance control, as a representative of indirect force control, has been adopted by mainstream collaborative arms such as UR, AUBO, and Franka due to its advantages of "simplest hardware and lowest cost." This approach converts end-effector force errors into displacement and adds them to the position trajectory, requiring only a six-dimensional force sensor on the wrist to achieve virtual compliance. Existing humanoid robots also typically deploy force sensors on the wrist and ankle, using admittance algorithms to "digest" contact forces into pose adjustments.

[0004] The prior art solutions most similar to this invention mainly include the following two categories:

[0005] Category 1: Admittance control method based on fixed parameters. Fixed stiffness and damping coefficients are commonly used offline in precision industrial assembly. This strategy can cope with batch scenarios where the gap is greater than 0.5mm and the workpieces have uniform rigidity, but it has obvious defects: (1) When the gap is less than 0.5mm or the workpiece is made of flexible material, the fixed stiffness is prone to jamming, scratching or loose insertion; (2) Production line changes require manual recalibration, and downtime is often measured in shifts, which is difficult to adapt to the rapid switching needs of multiple varieties in flexible manufacturing.

[0006] The second category is the parameter self-tuning admittance control method based on adaptive laws. To eliminate manual calibration, the academic community has proposed using stiffness and damping as time-varying matrices, estimating the environmental stiffness online through Kalman filtering or least squares, and refreshing the parameters in real time based on the Lyapunov function. However, this method has the following limitations: (1) It is highly dependent on the prior of the second-order contact model, but the contact stiffness of flexible thin-walled parts changes drastically and nonlinearly, and the model itself has structural errors; (2) Online estimation requires continuous excitation, but the "brief light touch" in shaft-hole assembly is difficult to meet the convergence condition, which can easily lead to estimation drift; (3) When the workpiece shape or lubrication changes, the stability boundary needs to be re-derived, otherwise it may cause self-excited oscillation, which is complicated to implement in engineering and has few practical cases.

[0007] In recent years, deep reinforcement learning has provided new ideas for compliant control, and the academic community has used algorithms such as DDPG and PPO to achieve end-to-end results in light contact tasks. However, existing research mostly focuses on position-level control, with force information only used as a penalty term in the reward function, and not truly integrated into the action space of the policy network, making it difficult to fully leverage the guiding role of force information in precision assembly.

[0008] In summary, existing technologies cannot simultaneously achieve the adaptability, stability, and ease of deployment of compliant control. There is an urgent need for an intelligent compliant control method that can dynamically adjust admittance parameters based on contact status and does not rely on a precise environmental model. Summary of the Invention

[0009] In precision shaft and hole assembly, the workpiece is small in size, has a small gap, and is made of soft material, making it difficult to accurately model contact dynamics. Furthermore, in flexible manufacturing processes involving multiple product batches, the physical properties of the workpiece change over time. Existing fixed-parameter admittance compliance control methods require repeated manual tuning of stiffness-damping parameters, which fails upon model changes. Over-contact can easily lead to jamming, while under-contact may result in incomplete insertion, with the success rate dropping sharply as the gap decreases. Existing adaptive law-based variable-parameter admittance control methods face difficulties in online estimation of contact dynamic parameters and complex proof of control stability when dealing with variable working conditions and thin-walled, easily deformable workpieces, hindering practical application. The purpose of this invention is to solve the above-mentioned problems in the existing technology and provide a reinforcement learning-based adaptive compliance force control method for robotic arms.

[0010] To achieve the above-mentioned objectives, the present invention specifically adopts the following technical solution:

[0011] This invention provides an adaptive compliant force control method for a robotic arm based on reinforcement learning, comprising the following steps:

[0012] The robotic arm-dexterous hand system first acquires the target spatial pose of the assembly hole on the target workpiece using a global RGBD camera placed outside the system. A policy network trained using reinforcement learning continuously generates adaptive parameters for each stage of the shaft-hole assembly process. These adaptive parameters, along with the target spatial pose, are input into a stage planner. The stage planner outputs the current desired pose in real time based on the progress of the shaft-hole assembly task. The difference between the desired pose and the actual pose in Cartesian space at the robotic arm tool end is calculated to obtain the pose deviation. This pose deviation is then input into a PD controller for trajectory tracking, and the system transmits the results to the policy network. The network generates PD control parameters for the PD controller online, performs preliminary trajectory correction on the pose deviation, and obtains the adjusted pose command. The policy network also outputs pose fine-tuning based on the pose deviation. Next, the dual-mode admittance dynamics controller receives the adaptive admittance parameters provided by the policy network and outputs compliant pose adjustment. The adjusted pose command, pose fine-tuning, and compliant pose adjustment are summed to obtain the Cartesian space control command. Finally, the Cartesian space control command is fed to the inverse kinematics algorithm of the robotic arm to solve for the joint end, obtaining the position control command for each joint, thus achieving compliant control.

[0013] As a preferred approach, the stage planner divides the entire shaft and hole assembly process into three stages: a free stage, a search stage, and an insertion stage. After parameter initialization, the system first enters the free stage. When the preset first stage switching condition is met, the control logic switches from the free stage to the search stage. When the preset second stage switching condition is met, the control logic switches from the search stage to the insertion stage. When the preset successful termination condition is met, the assembly is completed.

[0014] Preferably, in the free segment, the target workpiece held by the dexterous hand is in the air and reaches a preset height directly above the assembly hole at a preset speed without impact. The admittance dynamics controller is disabled or its stiffness parameters are kept within a preset stiffness range. The reinforcement learning strategy network outputs pose fine-tuning and PD control parameters to assist the PD controller in pose correction, without adjusting the stiffness parameters of the admittance dynamics controller. In the search segment, a parametric spiral is used to search for the assembly hole position, and the radius contraction rate and pressing speed are adjusted online through the strategy network. The PD controller and pose fine-tuning are used for trajectory tracking, and the admittance dynamics controller is added to achieve compliant and safe contact. In the insertion segment, the normal direction of the assembly hole is used as the Z-axis, and the X-axis and Y-axis are constructed with the tangential direction of the assembly hole. The three axes X, Y, and Z are perpendicular to each other, and the XY-axis is decoupled from the Z-axis. The Z-axis position control is frozen. The X-axis and Y-axis are kept compliantly inserted and aligned by the PD controller, reinforcement learning pose fine-tuning, and standard admittance control. Constant force admittance control is used to maintain the contact force tracking of the Z-axis at the end of the robotic arm.

[0015] Preferably, the first-stage switching condition is that the target workpiece reaches a preset height directly above the assembly hole, the tangential position error is less than or equal to a preset tangential position error threshold, and the Z-axis force component of the six-dimensional force sensor at the end of the robotic arm is less than a preset force threshold. The second-stage switching condition is that the target workpiece has entered the preset depth of the assembly hole, the tangential position error is less than or equal to a preset tangential position error threshold, and the Z-axis force component of the six-dimensional force sensor at the end of the robotic arm is greater than or equal to a preset force threshold. When the insertion depth is greater than a preset task successful insertion depth threshold, the Z-axis force error is less than a preset Z-axis force error threshold, and the assembly hole tangential position error is less than a preset tangential position error threshold, the successful termination condition is considered to have been met.

[0016] As a preferred embodiment, in the admittance force correction of the insertion segment, the tangential XY direction uses standard second-order admittance control, and the admittance parameters are output by the strategy network; the normal Z direction uses constant force admittance, and the admittance parameters are output by the strategy network, and the magnitude of the desired force on the Z axis is adjusted by the constant force adjustment parameters generated by the strategy network.

[0017] As a preferred approach, during the training of the policy network, a segmented reward method is used to evaluate the performance of the policy network at each stage and calculate the cumulative reward: In the free phase, the reward function is used to simultaneously ensure that the tangential position error is less than a preset position error threshold, the output amplitude of the reinforcement learning action space is less than a preset action change threshold, the external actual contact force is less than a preset contact force threshold, and a penalty for each step that encourages the policy network to complete quickly; the reward function in the search phase is based on the reward function in the free phase, with an additional insertion depth reward; in the insertion phase, the reward function is used to simultaneously ensure that the tangential position error is less than the position error threshold, the output amplitude of the reinforcement learning action space is less than the action change threshold, the normal force error is less than a preset normal force accuracy threshold, and the absolute value of the external actual contact force in the tangential direction is less than a preset contact force threshold, while retaining the insertion depth reward and the penalty for each step that encourages the policy network to complete quickly;

[0018] When a free segment completes its transition to the search segment, or when a search segment completes its transition to the insertion segment, or when the aforementioned successful termination condition is met, an additional preset amount of sparse reward is issued.

[0019] Preferably, the policy network consistently outputs action vectors of a fixed dimension. In the environment interaction layer during reinforcement learning training, invalid actions are identified based on valid stage markers. Invalid actions are then zeroed out using a combination of hard and soft constraints, thus achieving a variable action space mechanism. Invalid actions are those corresponding to adaptive parameters not used in the current stage. Hard constraints involve forcibly zeroing out invalid actions before feeding them into the control loop, while soft constraints involve setting a dimension penalty term in the reward function of each stage. The penalty term in this dimension guides the policy network to autonomously learn invalid actions in the current stage; This is the penalty coefficient; This is the set of indices for invalid actions in the current phase. Index for invalid actions; The size of the invalid action; during reinforcement learning training, the stage identifier is input into the policy network as part of the reinforcement learning observation space, so that the policy network can learn the conditional policy and output different types of parameters at different stages.

[0020] As a preferred approach, an action masking mechanism is employed in each stage of reinforcement learning training to set the action mask value of invalid actions to 0. Invalid actions are those corresponding to adaptive parameters that are not used in the current stage. During reinforcement learning training, stage identifiers are input into the policy network as part of the reinforcement learning observation space, enabling the policy network to learn conditional policies and output different types of parameters at different stages.

[0021] As a preferred approach, the adaptive parameters used in the current stage are taken as effective parameters. The effective parameters for the reinforcement learning action space output in each stage are as follows: In the free segment, the pose fine-tuning amount and PD control parameters are effective parameters; in the search segment, the pose fine-tuning amount, stiffness parameters, PD control parameters, radius contraction rate, and compression speed are effective parameters; in the insertion segment, the pose fine-tuning amount, stiffness parameters, PD control parameters, and constant force adjustment parameters are effective parameters.

[0022] As a preferred approach, the reinforcement learning observation space collects pose deviation, end-effector speed, external contact force, stage markers, and previous action in real time at each moment. This data is used to adjust the weights of the reinforcement learning network in real time, thereby finding the action strategy with the optimal reward.

[0023] Compared with the prior art, the present invention has the following advantages:

[0024] This invention incorporates the stiffness and damping parameters of the admittance dynamics controller, the six-dimensional pose fine-tuning in Cartesian space, the PD control parameters of the PD controller, and the adaptive parameters of the stage planner into the reinforcement learning action space. It also considers pose deviation, end-effector speed, and external contact force at each moment as states. A segmented adaptive insertion assembly logic is designed, and a staged reward function for reinforcement learning training is constructed. During simulation training and physical transfer, the policy network autonomously learns a global strategy of "when to be flexible, when to be rigid, how to search, and how to insert." During workpiece changeover, only online... With fine-tuning for tens of minutes, without any contact model or parameter estimation, it can maintain a high assembly success rate and low peak contact force (<5N) for tasks with high assembly precision requirements, such as small gaps of 0.05mm, easily deformable, and multi-material workpieces. It solves the problems of poor adaptability and long debugging time of existing fixed parameter admittance control methods in actual industrial environments, as well as the shortcomings of existing variable parameter admittance control methods based on adaptive laws, such as difficulty in online estimation of contact dynamic parameters and poor stability in variable working conditions. It completely eliminates the two major bottlenecks of manual calibration and model estimation, and realizes rapid switching of flexible production lines with "zero downtime". Attached Figure Description

[0025] Figure 1 This is a control block diagram of the method of the present invention;

[0026] Figure 2 The logic diagram for the stage planner;

[0027] Figure 3 Training curve for shaft hole force-controlled insertion task;

[0028] Figure 4 This is a graph evaluating the effectiveness of a strategy for slow insertion, showing the change of the XYZ triaxial force of the insertion segment over time.

[0029] Figure 5 This is a graph evaluating the effectiveness of a strategy to assess the change in lateral error of the X and Y axes over time during normal speed insertion. Detailed Implementation

[0030] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in various embodiments of the present invention can be combined accordingly without mutual conflict.

[0031] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0032] To address the problems of poor adaptability and long debugging time of constant-parameter admittance control methods in actual industrial environments, and the shortcomings of existing technologies such as difficulty in online estimation of contact dynamic parameters and poor stability of adaptive law-based variable-parameter admittance control methods under varying working conditions, this invention proposes an adaptive compliant force control method for robotic arms based on reinforcement learning. This method employs an integrated framework of "reinforcement learning-adaptive compliant force control" based on a robotic arm-dexterous hand-six-dimensional force / torque sensor system, such as... Figure 1 As shown, it includes:

[0033] The robotic arm-dexterous hand system first acquires the target spatial pose of the assembly hole on the target workpiece using a global RGBD camera placed outside the system. The policy network trained using reinforcement learning continuously generates adaptive parameters for each stage throughout the entire shaft-hole assembly process, and then correlates these adaptive parameters with the target space pose. The input is fed into the stage planner, which outputs the current desired pose in real time based on the progress of the shaft and hole assembly task. The desired pose is compared with the actual pose of the current robotic arm tool end in Cartesian space. The difference is calculated to obtain the pose deviation. ; to adjust the pose deviation The PD controller is input for trajectory tracking, and the PD control parameters are generated online through the policy network to address pose deviations. After initial trajectory correction, the adjusted pose command is obtained. ;in, and These are all PD control parameters of the PD controller, indicated by the superscript " " indicates taking the first derivative, i.e. for The first derivative; the policy network also considers the pose deviation. Output pose fine-tuning amount Next, the dual-mode admittance dynamics controller receives adaptive admittance parameters provided by the policy network. Internally, based on the current task progress, the admittance dynamics controller calculates the compliant pose adjustment amount used to correct the influence of contact force on the system using a mass-spring-damped second-order system model (standard second-order admittance model and constant force second-order admittance model). ; Adjust the pose command Posture fine adjustment amount Flexibility adjustment amount Summing yields the Cartesian space control command. Finally, the Cartesian space control command will be executed. The inverse kinematics algorithm of the robotic arm is used to solve the joint position control commands for each joint, thus achieving compliant control. Meanwhile, reinforcement learning is used to collect pose deviations in real time at each moment. End-effector speed External actual contact force The system uses stage identifiers and the previous action to adjust the weights of the reinforcement learning network in real time, thereby finding the action strategy with the optimal reward. It should be noted that the robotic arm-dexterous hand system described above is a system with a dexterous hand attached to the end effector of a robotic arm. In terms of control strategy, this invention adopts a hierarchical control architecture of "upper-level stage planner + lower-level variable-parameter admittance dynamics controller - RL dual closed loop," employing a four-level cascaded control method, including: stage planner - PD feedforward trajectory tracking - reinforcement learning motion fine-tuning - variable-parameter admittance compliant pose superposition - inverse kinematics calculation of joint control commands, a complete process of five parts. The PD controller, admittance dynamics controller, and stage planner are described below.

[0034] It should be noted that in this invention, the optimal PD control parameters of the PD controller are generated by the policy network. In this embodiment, to reduce the dimension of the reinforcement learning action space and maintain system stability, the position gain is... With speed gain Satisfying Relationships Critical damping is used, meaning that for the reinforcement learning action space, only the adaptive output is used. By referring to the parameters, you can obtain all the PD control parameters of the PD controller.

[0035] It should be noted that the stage planner of the present invention divides the entire shaft hole assembly process into three stages: free stage, search stage, and insertion stage. After parameter initialization, it first enters the free stage. When the preset first stage switching condition is met, the control logic switches from the free stage to the search stage. When the preset second stage switching condition is met, the control logic switches from the search stage to the insertion stage. When the preset successful termination condition is met, the assembly is completed.

[0036] In the free segment, the target workpiece held by the dexterous hand is in the air and is brought to a preset height directly above the assembly hole at a preset speed without impact. The admittance dynamics controller is disabled or its stiffness parameters are kept within a preset stiffness range. The reinforcement learning policy network outputs pose fine-tuning and PD control parameters to assist the PD controller in pose correction, without adjusting the stiffness parameters of the admittance dynamics controller. In the search segment, a parametric spiral is used to search for the assembly hole position, and the radius contraction rate and pressing speed are adjusted online through the policy network. The PD controller and pose fine-tuning are used for trajectory tracking, and the admittance dynamics controller is added to achieve compliant and safe contact. In the insertion segment, the normal direction of the assembly hole is used as the Z-axis, and the X-axis and Y-axis are constructed with the tangential direction of the assembly hole. The three axes X, Y, and Z are perpendicular to each other, and the XY-axis is decoupled from the Z-axis. The Z-axis position control is frozen. The X-axis and Y-axis are kept compliantly inserted and centered through the PD controller, reinforcement learning pose fine-tuning, and standard admittance control. Constant force admittance control is used to maintain the contact force tracking of the Z-axis at the end of the robotic arm.

[0037] Furthermore, the first-stage switching condition is that the target workpiece reaches a preset height directly above the assembly hole, the tangential position error is less than or equal to a preset tangential position error threshold, and the Z-axis force component of the six-dimensional force sensor at the end of the robotic arm is less than a preset force threshold. The second-stage switching condition is that the target workpiece has entered the preset depth of the assembly hole, the tangential position error is less than or equal to a preset tangential position error threshold, and the Z-axis force component of the six-dimensional force sensor at the end of the robotic arm is greater than or equal to a preset force threshold. When the insertion depth is greater than a preset task successful insertion depth threshold, the Z-axis force error is less than a preset Z-axis force error threshold, and the assembly hole tangential position error is less than a preset tangential position error threshold, the successful termination condition is considered to have been met.

[0038] The execution flow within each of the three stages of the stage planner will be explained in detail below.

[0039] Within the training loop of this embodiment, the environment's `step()` function is primarily responsible for the logic of interaction between the robotic arm-dexterity hand system and the simulation environment during training. It uses the MuJoCo API for signal reading and release control, and calculates the reward function and successful termination conditions. In the interaction logic, taking the shaft hole assembly task of "vertically inserting a 10cm long pin into a workpiece with a depth of 7cm" as an example, this invention focuses on maintaining the upper-level stage planner for shaft hole assembly. This planner divides the entire assembly process into three parts: a free segment, a search segment, and an insertion segment. It uses prior knowledge of staged insertion in shaft hole assembly to assist reinforcement learning training, and combines this with the self-exploration capability of the policy network to find the optimal upper-level planning implementation based on the prior knowledge. The process is as follows: Figure 2 As shown. In the control loop, the stage planner is located at... Figure 1On the far left, the desired pose for the current stage is output every 0.01 seconds for subsequent three-stage cascaded adaptive reinforcement learning admittance control. The transition criteria for the three stages use a dual threshold of "distance-force," calculated every control cycle (…). The stage flag is updated every 0.01 seconds. .

[0040] In the free segment of this embodiment, the target workpiece needs to reach a position 10mm directly above the orifice at a high speed without impact to prepare for the subsequent helical search. Touching the workpiece is not allowed during this stage. To avoid force noise disturbance, the admittance dynamics controller can be completely disabled, or its stiffness can be kept at an extremely low setting. The policy network outputs a small pose fine-tuning amount to assist the PD controller in pose correction, without adjusting the stiffness parameter. Furthermore, in the free segment, it is desired that the pose uses single-step interpolation.

[0041]

[0042] in, When the time step is in the free segment The expected pose at that time; Let be the robot's initial pose, and take a constant value; For the time step of the free segment; For the end effector of the robotic arm to reach the target position during the free segment. Total time; superscript Indicates transpose; This is the preset height position directly above the assembly opening.

[0043] In this embodiment, when the pin insertion end reaches 10mm above the hole (i.e. The tangential position error is less than or equal to 10 mm (i.e.) Furthermore, the Z-axis force component measured by the six-dimensional force sensor at the end of the robotic arm... If the value is less than 2N (not touched), it is considered that the "search area has been reached," satisfying the first-stage switching condition. The control logic then switches from the free segment to the search segment. The specific mathematical conditions are:

[0044]

[0045] in, For the first The stage at which each time step is located; Used to connect two or more conditions that need to be satisfied simultaneously; This refers to the tangential position error; This refers to the tangential position of the pin insertion end; The tangential position of the hole center; The insertion depth is calculated in real time; The height of the hole surface along the Z-axis; This represents the real-time height of the pin insertion end on the Z-axis.

[0046] In the search segment of this embodiment, basic parametric spiral search prior knowledge is employed. Time step From the moment the search segment begins, the baseline spiral equation is as follows:

[0047]

[0048]

[0049]

[0050] in, To progress over time The radius of a monotonically decreasing circular motion; The radius shrinkage rate; The initial circumference radius; To progress over time A monotonically increasing circumferential angle; The preset angular velocity of circular motion; Indicates time step as Position of the circular motion at that time; These are the position components in three directions; To progress over time The downward pressure speed increases monotonically.

[0051] In the spiral search, the radius contraction rate and the downward pressure speed are key parameters controlling the spiral's ability to slide down and contract its radius within the "beveled cone surface." If the radius contraction rate is too large, the radius descends too quickly, and the pin may hit the bottom before reaching the center of the hole, resulting in a hard collision; if the radius contraction rate is too small, the number of search cycles increases, wasting time. Similarly, an excessively high downward pressure speed may generate force spikes; an excessively low downward pressure speed slows down the cycle. Therefore, the strategy network of this invention uses actions... , The system allows for online adjustment of the radius shrinkage rate and pressing speed, enabling adaptive adjustment of the circumferential radius between 0-10mm and the pressing speed between 0.5-1.5mm / s. It automatically adapts to different chamfer angles, lubrication conditions, and workpiece hardness without manual instruction. Specifically, the online adjustment mechanism for the radius shrinkage rate and pressing speed is as follows:

[0052]

[0053]

[0054] in, To enhance the radius shrinkage rate under learning control; The action output by the policy network, used to adjust the radius shrinkage rate; To enhance the downward pressure speed under learning control; This is the action output by the policy network used to adjust the downward pressure speed.

[0055] The radius contraction rate and the downward pressing speed directly control the "tangential convergence speed" and the "normal feed speed." Reinforcement learning automatically learns the optimal combination through gradient signals, ensuring that the pin insertion end always stays in contact with the chamfered cone surface. The radius of motion monotonically decreases, and the insertion depth monotonically increases, achieving a spiral search that "contracts while sliding down." This method offers high sample efficiency and eliminates the need for recalibration when changing pin types.

[0056] Therefore, in the search segment, the time step is... Desired pose at time Represented as:

[0057]

[0058] in, Location of the orifice; These are the orifice position components in three directions; This is the distance in the depth direction between the end of the pin and the opening of the hole at the start of the search segment; To represent a quaternion pointing vertically downwards.

[0059] Furthermore, during the search phase, an admittance dynamics controller is enabled, using the standard second-order mass-spring-damped admittance dynamics equations to compliantly avoid any potential collisions. The system also employs reinforcement learning to output optimal stiffness parameters in real-time within the action space. The stiffness mapping range for this segment is:

[0060]

[0061] in, For the first Dimensional stiffness parameters; To enhance the learning control of the first Dimensional stiffness parameters; The output of the policy network is used to regulate the first... The action of the stiffness parameter; This is the dimensional index for the stiffness parameters. To reduce the dimensionality of the action space in reinforcement learning, compliant damping... Stiffness parameters of the corresponding dimension Seeking, satisfying relation, The damping ratio; For the first The inertia of a dimension.

[0062] Because the tip of the target workpiece (e.g., the end of a pin) is close to the chamfer of the hole to be installed, but there is an unknown offset of 0-0.4mm at the center of the hole, continuing to press down could easily cause the inserted workpiece to press against the chamfer, creating a force peak and causing damage. The spiral search, sliding down the chamfered conical surface while shrinking the radius, can reduce the tangential error to less than 1mm, while maintaining the Z-axis force component without making it too large, thus avoiding scratches between workpieces.

[0063] In this embodiment, when the pin has entered the hole to a depth of less than 1mm, the tangential position error with the hole opening is less than or equal to 1mm, and the Z-axis force component is greater than or equal to 2N (the chamfer has been touched), it is considered to have "entered the guide zone," satisfying the second-stage switching conditions. The control logic then switches from the search segment to the insertion segment. The specific mathematical conditions are:

[0064]

[0065] In the insertion section, the pin has entered the chamfered guide zone. The tangential (XY) fit tolerance is only 0–0.05mm, requiring stable and smooth contact with the hole wall to prevent scratches between workpieces. The normal (Z) direction needs to overcome friction; if traditional position control is used for pressing, force spikes are likely to occur due to changes in workpiece rigidity. This invention adopts a directional decoupling strategy of "lateral position-impedance + vertical constant force-admittance," which involves the entire process from the generation of the desired pose in the outer loop to the issuance of the final position loop command: in the tangential direction, the strategy network only adjusts the translational stiffness controlled by admittance, while the rotational stiffness remains fixed; in the normal direction, no position command is given, but a constant force is used for pressing, and the strategy network fine-tunes the Z-axis force based on the frictional characteristics of the workpiece surface. Figure 1 From the control block diagram flow, the settings for the insertion segment are as follows:

[0066] 1) Desired trajectory generation (direction decoupling)

[0067] The desired trajectory described above is formed by the target position of the inserted segment in the XY direction. In generating the desired trajectory, the center position of the insertion hole is locked in the tangential XY direction, i.e. In the normal Z direction, the desired pose is no longer tracked. The position is generated by an admittance constant force, which is then used as a substitute. Ultimately, the desired pose is determined. It can be represented as , This is a unit quaternion used to represent vertical insertion aligned with the orifice. Wherein, This indicates the target position of the inserted segment in the tangential XY direction. This indicates the tangential position of the hole center, i.e., the position of the assembly hole center in the tangential XY direction; This indicates the target position of the inserted segment in the normal Z direction.

[0068] 2) PD feedforward control

[0069] In the control loop of the PD controller, only the pose tracking control of the five dimensions of tangential XY and rotation (RX, RY, RZ) is performed. The normal Z direction is not subject to PD feedforward control and does not participate in position tracking.

[0070] For the tangential XY:

[0071]

[0072] in, The pose command adjusted in the XY direction; For the XY direction parameter; This refers to the tangential position of the pin insertion end, i.e., the actual position of the pin insertion end in the XY direction; For the XY direction Parameters, take the critical damping; This represents the actual speed of the pin insertion end in the XY direction.

[0073] For rotations (RX, RY, RZ):

[0074]

[0075] in, The pose command adjusted in three rotational directions; For three rotational directions parameter; For three rotational directions Parameters, take the critical damping; This refers to the actual position of the pin insertion end in the three rotational directions; The desired position of the pin insertion end in three rotational directions; The current angular velocity in the three rotational directions.

[0076] Therefore, the adjusted pose command finally output by the PD controller is expressed as:

[0077]

[0078] 3) Enhance learning fine-tuning

[0079] Reinforcement learning motion space output pose fine-tuning and the output of the PD controller Summing yields intermediate variables. .

[0080] 4) Admittance correction

[0081] Tangential XY directions use standard second-order admittance control, with admittance parameters output by the policy network:

[0082]

[0083] in, The moment of inertia is in the tangential direction; The acceleration is in the tangential direction; For tangential compliance damping; External actual contact force in the tangential direction; stiffness parameter in the tangential direction. By reinforcing the action space, the corresponding optimal value is output in real time, and the mapping range is as follows:

[0084]

[0085] in, To enhance stiffness parameters under learning control ; The output of the policy network, used to adjust the stiffness parameters. The movement is relatively stiff here, in order to make the centering more accurate.

[0086] when =0.01s, perform discrete update:

[0087]

[0088]

[0089] in, The tangential target velocity generated for the admittance dynamics controller; the superscript -1 indicates inversion; This refers to the compliant pose adjustment amount in the tangential direction.

[0090] The normal Z-axis uses constant force admittance, with admittance parameters output by the policy network and adjusted by constant force adjustment parameters generated through reinforcement learning to control the magnitude of the desired Z-axis force.

[0091]

[0092]

[0093]

[0094] in, The moment of inertia is in the normal direction; This is the acceleration in the normal direction; For compliant damping in the normal direction; This represents the actual velocity of the pin insertion end in the Z direction; This represents the actual position of the pin insertion end in the Z direction. The external actual contact force is in the normal direction; The desired force along the Z-axis, By using a first-order low-pass filter, the target Z-axis force jump amplitude caused by reinforcement learning adaptation can be slowed down, thus avoiding the generation of large instantaneous errors that affect the policy network update. The first The expected Z-axis force at each time step; The output of the policy network is used to adjust the desired force along the Z-axis. The movement is designed to adapt to the surface friction of different workpieces, enabling smoother insertion; The initial value of the desired force along the Z-axis; the stiffness parameter in the normal direction. By using reinforcement learning, the action space is output in real time, and its corresponding optimal value is output. The mapping range is as follows:

[0095]

[0096] in, To enhance stiffness parameters under learning control ; The output of the policy network, used to adjust the stiffness parameters. The action. The stiffness here is relatively low, used to achieve compliant pressing and prevent scratching.

[0097] when =0.01s, perform discrete update:

[0098]

[0099]

[0100] In the formula, The normal target velocity generated for the admittance dynamics controller; This is the amount of compliant posture adjustment in the normal direction.

[0101] The rotational directions (RX, RY, RZ) are controlled using conventional second-order admittance, with fixed admittance parameters that are not regulated by reinforcement learning, thus reducing complexity. Specifically, in the insertion segment, since the pin has entered the chamfered guide zone and its attitude is almost vertical, the stiffness parameters in the rotational directions are fixed. This eliminates the need for reinforcement learning for dynamic adjustment, reducing training difficulty. In this embodiment, the three elements of the rotational direction stiffness parameter... Both remain constant at 45 N / m, with compliant damping in the direction of rotation. and Still satisfied , relation. Inertia in the direction of rotation. Compliant pose adjustment in the direction of rotation. The solution is the same as that for the compliant pose adjustment in the tangential direction, and will not be repeated here. In summary, the compliant pose adjustment is expressed as: .

[0102] 5) Inner ring synthesis

[0103] Through the above four steps, the policy network outputs the pose fine-tuning amount. The six-dimensional force sensor at the end effector of the robotic arm inputs the six-dimensional contact force / torque, after sensor calibration and gravity compensation, to the admittance dynamics controller. Simultaneously, the strategy network provides the admittance dynamics controller with the optimal admittance parameters in real time, including stiffness parameters and the damping matrix. Internally, based on the current task progress, the admittance dynamics controller calculates the compliant posture adjustment amount used to correct the impact of the contact force on the system using a mass-spring-damped second-order system model (standard second-order admittance model and constant-force second-order admittance model). and the previous and The final Cartesian space control command is obtained by superposition. .

[0104] Finally, The inverse kinematics algorithm of the robotic arm calculates the position control commands for each joint, and the angle control commands are executed through the MuJoCo simulation interface or the real robot interface. .in, Indicates the actual external contact force; Indicates admittance dynamics controller; This represents the inverse kinematics algorithm for the robotic arm. During the simulation training phase, the policy network can be trained using the above process. Then, the same control architecture and reinforcement learning agent can be deployed on a real robot to achieve the control effect of this invention.

[0105] In this embodiment, when the insertion depth calculated in real time is simultaneously satisfied... The insertion depth of the task is greater than the preset threshold (6.5cm), and the Z-axis force error is greater than the threshold value. Less than the preset Z-axis force error threshold (0.5N), assembly hole tangential position error When the error is less than the preset tangential position error threshold (0.5mm), the termination condition is considered to have been successfully met.

[0106]

[0107] It should be noted that, in the policy network training process of this invention, a segmented reward method is used to evaluate the execution of the policy network at each stage and calculate the cumulative reward. In the free segment, the reward function is used to simultaneously ensure that the tangential position error is less than a preset position error threshold, the output amplitude of the reinforcement learning action space is less than a preset action change threshold, the external actual contact force is less than a preset contact force threshold, and a penalty for each step that encourages the policy network to complete quickly is applied. The reward function in the search segment is based on the reward function in the free segment, with an additional insertion depth reward. In the insertion segment, the reward function is used to simultaneously ensure that the tangential position error is less than the position error threshold, the output amplitude of the reinforcement learning action space is less than the action change threshold, the normal force error is less than a preset normal force accuracy threshold, and the absolute value of the external actual contact force in the tangential direction is less than a preset contact force threshold, while retaining the insertion depth reward and the penalty for each step that encourages the policy network to complete quickly is applied.

[0108] Furthermore, during reinforcement learning training, an additional +10 sparse reward is awarded when the free segment completes and enters the search segment, or when the search segment completes and enters the insertion segment; and an additional +20 sparse reward is awarded when the above-mentioned successful termination conditions are met.

[0109] The reward function settings for the three stages are explained in detail below. First, the linear normalization function is defined:

[0110]

[0111] in, For function variables; For variables The L2 norm; The maximum value of the variable; To obtain the minimum value.

[0112] In this embodiment, the reward function of the free segment The goal is to achieve rapid, collision-free approach while maintaining an external contact force of less than 2N. This involves tangential positional accuracy, range of motion space variation, magnitude of external contact force, and step size penalty, expressed as:

[0113]

[0114] in, For the free segment reward weight, , , ; The position error is in the tangential direction; the position error in the normal direction is negligible. As the position error threshold, in the free segment of this embodiment, It is 0.02m; Output amplitude for motion space; As the threshold for action change, in the free segment of this embodiment, Setting it to 0.02 limits the output changes of the action from being too drastic; This refers to the actual external contact force. As the contact force threshold, in the free segment of this embodiment, Set to 2N; A small penalty is applied at each step to encourage the strategy network to complete the task quickly. Once the free phase is completed, the search phase begins, and a sparse reward of +10 is awarded.

[0115] In this embodiment, the reward function of the search segment Specifically, this involves tangential positional accuracy, range of motion space variation, magnitude of actual external contact force, step size penalty, and insertion depth, expressed as:

[0116]

[0117] in: For search segment reward weight, , , , In the search segment of this embodiment, It is 0.002m. Set to 0.02, Set to 5N; As the insertion depth threshold, in the search segment of this embodiment, The value is 6.5cm. Here, before calculating the reward function for this segment, the real-time tangential position error needs to be calculated first. ,if Larger than the radius of the circular assembly hole Then the insertion depth This prevents the policy network from incorrectly inserting into the orifice during the search segment, only when... Calculate according to the above process Once the search segment is complete, proceed to the insertion segment and award a sparse bonus of +10.

[0118] In this embodiment, the reward function of the insertion segment The goal is to achieve tangential alignment and constant force insertion in the normal direction, with an insertion depth greater than 65mm (assuming a complete workpiece depth of 70mm, 65mm is set here to meet the requirement). Specifically, this involves tangential positional accuracy, range of motion space variation, Z-axis force tracking accuracy, external actual contact force error in the tangential direction, step size penalty, and insertion depth.

[0119]

[0120] in, For the insertion segment reward weight, , , , , In the inserted segment of this embodiment, It is 0.001m. Set to 0.02, Set to 6.5cm; As the normal force accuracy threshold, in the inserted segment of this embodiment, Set to 0.5N; This refers to the error in the normal force. This represents the absolute value of the actual external contact force in the tangential direction; As the contact force threshold, in the inserted segment of this embodiment, Set to 1N.

[0121] Once the above termination conditions are met, the entire assembly task is considered successfully completed, the episode terminates, and a +20 sparse reward is awarded. Through the refined reward function settings described above, the policy network can be guided to autonomously learn and complete high-precision shaft and hole assembly tasks.

[0122] It should be noted that during the phased assembly process, the dimensions of the reinforcement learning action space change as the task progresses. This invention employs an action mask mechanism to implement a variable action space. Specifically, the action space remains constant at 21 dimensions, and action masks of different dimensions are used based on the phase identifiers (FREE, SEARCH, INSERT) of different assembly segments. (1 = valid, 0 = forced to zero), the action vector is element-wise multiplied with the original action vector output by the policy network to mask invalid action dimensions in the current stage, setting their values ​​to 0 and excluding them from policy gradient updates. This achieves the versatility of the same policy network in handling multi-stage tasks, avoids policy confusion, and improves sample efficiency. The Mask table is shown below, illustrating the validity of different parts of the action space at different stages:

[0123] Table 1. Mask Table

[0124] Furthermore, the action masking mechanism is applicable to various reinforcement learning algorithms. For discrete action space types, the mask is generally applied to the logits layer of the policy network, setting the logits of the position corresponding to the invalid action to negative infinity before softmax sampling. For continuous action space algorithms, the mask is applied to the action output layer, forcibly setting the invalid dimension to zero and shielding its gradient backpropagation. By adjusting the output probability distribution or action value distribution of the policy network, the probability of the invalid action being sampled or output tends to zero.

[0125] As an alternative feasible solution, considering the interface limitations of existing open-source reinforcement learning frameworks for implementing continuous action space algorithms, this invention also proposes an action space constraint method based on stage penalties. Specifically, the policy network always outputs action vectors of a fixed dimension. In the environment interaction layer during reinforcement learning training, invalid actions are identified based on valid stage markers, and invalid actions are zeroed out through a combination of hard and soft constraints. Invalid actions are those corresponding to adaptive parameters not used in the current stage. The adaptive parameters used in the current stage are taken as valid parameters. The valid parameters output by the reinforcement learning action space at each stage are as follows: In the free stage, pose fine-tuning and PD control parameters are valid parameters; in the search stage, pose fine-tuning, stiffness parameters, PD control parameters, radius contraction rate, and compression speed are valid parameters; in the insertion stage, pose fine-tuning, stiffness parameters, PD control parameters, and constant force adjustment parameters are valid parameters. Hard constraints involve forcibly zeroing out invalid actions before feeding them into the control loop, while soft constraints involve setting a dimension penalty term in the reward function at each stage. The penalty term in this dimension guides the policy network to autonomously learn invalid actions in the current stage; This is the penalty coefficient; This is the set of indices for invalid actions in the current phase. Index for invalid actions; The size of the invalid action.

[0126] Both of these approaches require inputting phase IDs (FREE, SEARCH, INSERT) as part of the reinforcement learning observation space into the policy network to learn conditional policies. This means outputting different types of adaptive parameters at different stages and selecting corresponding reward function calculation methods based on different stages. The phase IDs are embedded into the observation vector using one-hot encoding or discrete integer encoding, and the stage switching logic is based on a preset physical threshold. Both approaches achieve the versatility of the same policy network in handling multi-stage tasks, avoiding policy confusion and improving task performance.

[0127] It should be noted that, to improve the transfer effect from simulation training strategy to real control strategy, in addition to the conventional sim2real method, a domain randomization method can be used for training based on this invention. Parameters such as workpiece stiffness, friction, shaft-hole assembly tolerances, and the initial insertion pose of the robotic arm are randomized within a certain region to train a sufficiently robust control strategy. After transfer to a real robot, it is adaptable to different workpiece characteristics and operating environments. After workpiece change, only a few dozen rounds of online training on the real robot are needed to adjust to the optimal strategy for the current working conditions.

[0128] Furthermore, to highlight the core of end-effector force control and reinforcement learning-based parameter self-learning, this invention does not involve the grasping stage. This invention assumes that the workpiece is initially grasped by the dexterous hand in a fixed posture, and that the finger joints remain constant throughout the subsequent assembly process. The policy network only controls the 6 degrees of freedom of the arm and its adaptive parameters, without outputting finger movements. If it is necessary to extend this to the entire grasp-transfer-assembly task chain, the dexterous hand's active degrees of freedom output can be enabled in the reinforcement learning action space.

[0129] It should also be noted that this embodiment further elaborates on the additional details involved in the implementation process of the above solution.

[0130] 1) Analytical IK: Based on the geometric closed solution of Aobo i5H, the inverse solution calculation is completed in about 0.08ms, ensuring the real-time performance of the control.

[0131] 2) External force compensation for six-dimensional force sensor: By calculating the load mass of the back-end entity of the six-dimensional force sensor in the recursive MuJoCo model, the static gravity, dynamic inertial force, Coriolis force and torque caused by the load are calculated by the principle of dynamics. Real-time compensation is performed through the center of gravity, and a first-order low-pass filter is added to remove high-frequency noise.

[0132] 3) Safety Layer: Includes three thresholds for force, velocity, and configuration. Exceeding any threshold triggers a violation training interruption and incurs a penalty. It includes self-collision penalties: 1 point is penalized for each collision between arm-link, arm-link-dexterity hand, arm-link-tabletop, or dexterity hand-tabletop. It also includes singularity detection and quaternion validity checks; if a violation occurs, the current joint positions are used as protection.

[0133] The following demonstrates the technical effectiveness of the reinforcement learning-based adaptive compliant force control method for robotic arms in the preferred implementation described above. The method of this invention was applied to train a six-axis robot on a shaft hole force control insertion task, for a total of 2000 rounds. The training curve is shown below. Figure 3 As shown, the cumulative reward for each round gradually increases with the number of training rounds, indicating that the training scheme is feasible; at the end of training, the agent frequently obtains sparse rewards for task completion, proving that the policy network gradually converges and can successfully execute the task.

[0134] Experiments were conducted using the trained policy network to evaluate its effectiveness. The changes in the XYZ triaxial forces over time were recorded during the slow insertion of the robotic arm in the insertion segment. Figure 4 As shown, the actual contact force in the XY direction remains less than 1N, indicating that the end pin experiences minimal force on the insertion tangential surface, with no scraping or jamming. The pin is accurately aligned throughout, demonstrating good force control. The error between the actual contact force in the Z direction and the expected contact force controlled by RL remains small, indicating good tracking effect of the constant force admittance in the Z direction. The XY lateral error of the distance between the insertion end of the robotic arm pin and the center of the hole is recorded, as follows: Figure 5 As shown, the free segment error decreases rapidly as the robotic arm moves from its initial position to directly above the hole; the lateral error of the search segment decreases further as the search progresses until it inserts into the center of the hole; and finally, the insertion segment maintains a minimal lateral error as it continues to insert deeper until the task is completed.

[0135] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A method for adaptive compliant force control of a robotic arm based on reinforcement learning, characterized in that, Includes the following steps: The robotic arm-dexterous hand system first acquires the target spatial pose of the assembly hole of the target workpiece using a global RGBD camera placed outside the system. A policy network trained using reinforcement learning continuously generates adaptive parameters for each stage throughout the entire shaft-hole assembly process. These adaptive parameters, along with the target spatial pose, are input into a stage planner. The stage planner outputs the current desired pose in real time based on the progress of the shaft-hole assembly task. The difference between the desired pose and the actual pose in Cartesian space at the current robotic arm tool end is calculated to obtain the pose deviation. The pose deviation is input into the PD controller for trajectory tracking, and the PD control parameters of the PD controller are generated online through the policy network. After preliminary trajectory correction of the pose deviation, the adjusted pose command is obtained. The strategy network simultaneously outputs pose fine-tuning based on pose deviation. Next, the dual-mode admittance dynamics controller receives the adaptive admittance parameters provided by the strategy network and outputs compliant pose adjustment. The adjusted pose command, pose fine-tuning, and compliant pose adjustment are summed to obtain the Cartesian space control command. Finally, the Cartesian space control command is fed to the inverse kinematics algorithm of the robotic arm to solve for the joint end, thereby obtaining the position control command for each joint and achieving compliant control.

2. The adaptive compliant force control method for a robotic arm based on reinforcement learning as described in claim 1, characterized in that, The stage planner divides the entire shaft and hole assembly process into three stages: free stage, search stage, and insertion stage. After parameter initialization, it first enters the free stage. When the preset first stage switching condition is met, the control logic switches from the free stage to the search stage. When the preset second stage switching condition is met, the control logic switches from the search stage to the insertion stage. When the preset successful termination condition is met, the assembly is completed.

3. The adaptive compliant force control method for a robotic arm based on reinforcement learning as described in claim 2, characterized in that, In the free segment, the target workpiece held by the dexterous hand is in the air and is brought to a preset height directly above the assembly hole at a preset speed without impact. The admittance dynamics controller is disabled or its stiffness parameters are kept within a preset stiffness range. The reinforcement learning policy network outputs pose fine-tuning and PD control parameters to assist the PD controller in pose correction, without adjusting the stiffness parameters of the admittance dynamics controller. In the search segment, a parametric spiral is used to search for the assembly hole position, and the radius contraction rate and pressing speed are adjusted online through the policy network. The PD controller and pose fine-tuning are used for trajectory tracking, and the admittance dynamics controller is added to achieve compliant and safe contact. In the insertion segment, the normal direction of the assembly hole is used as the Z-axis, and the X-axis and Y-axis are constructed with the tangential direction of the assembly hole. The three axes X, Y, and Z are perpendicular to each other, and the XY-axis is decoupled from the Z-axis. The Z-axis position control is frozen. The X-axis and Y-axis are kept compliantly inserted and centered through the PD controller, reinforcement learning pose fine-tuning, and standard admittance control. Constant force admittance control is used to maintain the contact force tracking of the Z-axis at the end of the robotic arm.

4. The adaptive compliant force control method for a robotic arm based on reinforcement learning as described in claim 3, characterized in that, The first stage switching condition is that the target workpiece reaches a preset height directly above the assembly hole, the tangential position error is less than or equal to the preset tangential position error threshold, and the Z-axis force component of the six-dimensional force sensor at the end of the robotic arm is less than the preset force threshold. The second-stage switching conditions are that the target workpiece has entered the preset depth of the assembly hole, the tangential position error is less than or equal to the preset tangential position error threshold, and the Z-axis force component of the six-dimensional force sensor at the end of the robotic arm is greater than or equal to the preset force threshold. When the following conditions are met simultaneously: the insertion depth is greater than the preset task successful insertion depth threshold, the Z-axis force error is less than the preset Z-axis force error threshold, and the assembly hole tangential position error is less than the preset tangential position error threshold, the successful termination condition is considered to have been met.

5. The adaptive compliant force control method for a robotic arm based on reinforcement learning as described in claim 3, characterized in that, In the admittance correction of the insertion segment, the tangential XY direction uses standard second-order admittance control, and the admittance parameters are output by the policy network; the normal Z direction uses constant force admittance, and the admittance parameters are output by the policy network, and the magnitude of the desired force on the Z axis is adjusted by the constant force adjustment parameters generated by the policy network.

6. The adaptive compliant force control method for a robotic arm based on reinforcement learning as described in claim 4, characterized in that, During the training of the policy network, a segmented reward method is used to evaluate the performance of the policy network at each stage and calculate the cumulative reward: In the free stage, the reward function is used to simultaneously ensure that the tangential position error is less than a preset position error threshold, the output amplitude of the reinforcement learning action space is less than a preset action change threshold, the external actual contact force is less than a preset contact force threshold, and a penalty for each step that encourages the policy network to complete quickly; the reward function in the search stage is based on the reward function in the free stage, with an additional insertion depth reward; in the insertion stage, the reward function is used to simultaneously ensure that the tangential position error is less than the position error threshold, the output amplitude of the reinforcement learning action space is less than the action change threshold, the normal force error is less than a preset normal force accuracy threshold, the absolute value of the external actual contact force in the tangential direction is less than a preset contact force threshold, and retains the insertion depth reward and the penalty for each step that encourages the policy network to complete quickly; When a free segment completes its transition to the search segment, or when a search segment completes its transition to the insertion segment, or when the aforementioned successful termination condition is met, an additional preset amount of sparse reward is issued.

7. The adaptive compliant force control method for a robotic arm based on reinforcement learning as described in claim 6, characterized in that, The policy network consistently outputs fixed-dimensional action vectors. During reinforcement learning training, the environment interaction layer identifies invalid actions based on valid stage markers and uses a combination of hard and soft constraints to zero out invalid actions, thus achieving a variable action space mechanism. Invalid actions are those corresponding to adaptive parameters not used in the current stage. Hard constraints involve forcibly zeroing out invalid actions before feeding them into the control loop, while soft constraints involve setting dimension penalty terms in the reward function of each stage. The penalty term in this dimension guides the policy network to autonomously learn invalid actions in the current stage; This is the penalty coefficient; This is the set of indices for invalid actions in the current phase. Index for invalid actions; The size of the invalid action; during reinforcement learning training, the stage identifier is input into the policy network as part of the reinforcement learning observation space, so that the policy network can learn the conditional policy and output different types of parameters at different stages.

8. The adaptive compliant force control method for a robotic arm based on reinforcement learning as described in claim 6, characterized in that, In each stage of reinforcement learning training, an action masking mechanism is used to set the action mask value of invalid actions to 0. Invalid actions are those corresponding to adaptive parameters that are not used in the current stage. During reinforcement learning training, the stage identifier is input into the policy network as a component of the reinforcement learning observation space, enabling the policy network to learn conditional policies and output different types of parameters at different stages.

9. The reinforcement learning-based adaptive compliant force control method for robotic arms as described in any one of claims 7 or 8, characterized in that, Using the adaptive parameters used in the current stage as valid parameters, the valid parameters for the reinforcement learning action space output in each stage are as follows: In the free segment, the pose fine-tuning amount and PD control parameters are valid parameters; in the search segment, the pose fine-tuning amount, stiffness parameters, PD control parameters, radius contraction rate, and compression speed are valid parameters; in the insertion segment, the pose fine-tuning amount, stiffness parameters, PD control parameters, and constant force adjustment parameters are valid parameters.

10. The adaptive compliant force control method for a robotic arm based on reinforcement learning as described in claim 6, characterized in that, The reinforcement learning observation space collects pose deviation, end-effector speed, external contact force, stage markers, and previous action in real time at each moment. This data is used to adjust the weights of the reinforcement learning network in real time, thereby finding the action strategy with the optimal reward.

Images