Reinforcement learning control method for multi-repetitive-task parallel robot in complex environment

Abstract

The invention provides a reinforcement learning control method for a multi-repetitive-task parallel robot in a complex environment, and belongs to the technical field of robot application. The technical problems that in a complex environment, a multi-repeated-task parallel robot is insufficient in control precision and weak in self-adaptive capacity are solved. According to the technical scheme, the method comprises the following steps that 1, a kinetic model is established; 2, analyzing and updating a friction compensation strategy, and designing a model reference adaptive control method to dynamically adjust the friction compensation strategy; 3, designing a strengthening force tracking controller; step 4, fusing the deep Q network and a model reference adaptive control method; and 5, building a real complex environment with dynamic change. The method has the advantages that the system can adapt to different operation conditions and environment changes by dynamically adjusting the control strategy, and the robustness and stability of the system are improved.

Description

Technical Field

[0001] The present invention relates to the field of robot technology application technology, and in particular to a reinforcement learning control method for a multi-repetitive task parallel robot in a complex environment. Background Art

[0002] In modern industrial production, especially in the fields of automobile manufacturing and electronic product assembly, parallel robots are widely used in high-precision component assembly and processing tasks. However, robots in this industry face many challenges. In complex production environments, temperature, humidity and other factors affect the contact force between the parallel robot and the workpiece, thereby affecting the effect of trajectory tracking and friction compensation. Robots not only need to handle workpieces of various shapes, materials and sizes, but also various obstacles and interference factors. How to enable the robot to accurately track the predetermined trajectory and perform effective friction compensation during the contact process to ensure the precise assembly and processing quality of the parts has become a difficult problem. In repetitive assembly and processing tasks, traditional iterative learning control can gradually improve the operation accuracy and consistency of the robot in complex environments by repeatedly optimizing the control strategy, but additional compensation mechanisms are required when facing changes in friction, which can also cause tracking errors and system instability. In actual environments, the system will be affected by factors such as initial value changes, time lags, and external disturbances, resulting in the system not having strict repeatability, which makes traditional iterative learning control ineffective in these cases. In addition, in the face of complex and dynamically changing environments, traditional iterative learning control cannot effectively adjust the strategy, thereby limiting its application effect in complex environments.

[0003] The paper "Sliding Mode Adaptive Iterative Learning Control of 2R1T Parallel Robot" proposes a sliding mode-adaptive-iterative learning control scheme to solve the repetitive machining trajectory tracking accuracy problem of a parallel robot with two rotational and one translational degrees of freedom. This method has two shortcomings: (1) When facing a complex environment, the nonlinearity and dynamic characteristics of the system make parameter adjustment difficult, resulting in the controller being unable to quickly adapt to environmental changes; (2) This control method has high requirements for the accuracy of the system model. In a complex environment, the robot will encounter many unknown dynamic changes and uncertainties, which makes accurate modeling difficult, thus affecting the effect of the control algorithm.

[0004] The paper "Research on Iterative Learning Control and Repetitive Control of Robots" designs a reasonable application of PID controller, discrete delay internal model repetitive control, and optimized linear time-varying repetitive control system for robot multi-repetitive tasks. This method has two shortcomings: (1) The control method converges slowly in complex environments, especially when the task environment changes greatly. Each iteration depends on the result of the previous one, and the dynamic changes of the environment make the optimization process of the control strategy complicated and slow. (2) The iterative learning process of this control method requires a lot of data processing and calculation. Especially when the environment complexity is high, the computational burden will increase significantly, which will affect the real-time performance and response speed of the system.

[0005] How to solve the above technical problems is the subject faced by the present invention. Summary of the invention

[0006] The purpose of the present invention is to provide a reinforced learning control method for a multi-repetitive task parallel robot in a complex environment, which solves the technical problems of insufficient control accuracy and weak adaptability of the multi-repetitive task parallel robot in a complex environment. The control method proposes an innovative parallel robot control strategy, which integrates an enhanced iterative friction learning compensation controller and a deep Q network DQN model reference adaptive control method to significantly improve the control accuracy and adaptability of the robot in performing multi-repetitive tasks in a complex environment; firstly, an enhanced iterative friction learning compensation controller is designed, which continuously analyzes and optimizes the friction compensation strategy using an iterative learning module, and dynamically adjusts the compensation strategy according to the tracking error through a model reference adaptive control method, thereby achieving higher friction compensation accuracy in multi-repetitive tasks; secondly, an enhanced force tracking controller is proposed, which measures the contact force in real time through a force sensor module, and adjusts the corresponding strategy using a force tracking control module to achieve precise control of mechanical properties and good compliance; finally, the present invention integrates DQN with the model reference adaptive control method to form an overall optimization control scheme, and continuously optimizes the control strategy using a deep network Q to cope with environmental changes and task requirements. This comprehensive approach is expected to significantly improve the robot's execution efficiency and operational performance in multiple repetitive tasks, providing an efficient and reliable solution to complex environmental challenges in practical applications.

[0007] The inventive idea of ​​the present invention is as follows: the present invention proposes a systematic strategy, including constructing a dynamically changing test environment in a simulation platform to simulate various challenges in reality. The environment includes movable obstacles, changing working conditions and uncertain environmental factors, aiming to comprehensively evaluate the adaptability and robustness of the robot's control strategy when facing these complex situations. In order to provide accurate control data, the parallel robot is equipped with advanced sensors and data acquisition equipment, which can record the robot's motion data, torque data and environmental changes in real time. These data will be used to perform multiple repetitive tasks and record the detailed execution of each task, including key performance indicators such as task completion time, control accuracy and force tracking error. By verifying the optimized control strategy in the actual application environment and performing multiple rounds of iterative improvements based on the feedback data in the actual operation, the present invention is committed to gradually improving the accuracy of the control strategy, the efficiency of task execution and the overall stability of the system. This method provides an efficient and reliable solution to ensure the excellent performance of the robot in a complex environment, and provides strong support for the variable tasks and environmental challenges in practical applications.

[0008] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is specifically: a reinforcement learning control method for a multi-repetitive task parallel robot in a complex environment, comprising the following steps:

[0009] Step 1: Establish a dynamic model of the parallel robot that comprehensively considers factors such as joint friction, load, and temperature change, and identify the friction characteristics of the parallel robot under different working conditions based on the model;

[0010] Step 2: Design an enhanced iterative friction learning compensation controller. The iterative learning module is used to analyze and update the friction compensation strategy, and the model reference adaptive control method is designed according to the generated tracking error to dynamically adjust the friction compensation strategy, thereby improving the control accuracy.

[0011] Step 3: Design an enhanced force tracking controller, analyze the mechanical characteristics of the parallel robot in performing multiple repetitive tasks, measure the contact force in real time through the force sensor module, and adjust the corresponding strategy with the help of the force tracking control module to achieve accurate and compliant force tracking and compliance control;

[0012] Step 4: Based on steps 2) and 3), the deep Q network and the model reference adaptive control method are integrated to realize the overall optimization control of the parallel robot, continuously optimize the execution efficiency of multiple repetitive tasks in complex environments, and dynamically adjust the control strategy according to different interactive tasks and environmental changes, thereby improving the system's adaptability and stability;

[0013] Step 5: Build a real complex environment with dynamic changes and configure the monitoring function of the data acquisition and analysis module to evaluate and optimize the enhanced learning control accuracy and adaptability of the multi-task parallel robot in a complex environment.

[0014] Furthermore, in step 1, a dynamic model of the parallel robot is established which comprehensively considers the influencing factors such as joint friction, load and temperature change:

[0015]

[0016] Where B(q) is the robot inertia matrix, is the Coriolis vector, g(q) is the gravity vector, is the joint friction vector, q is the joint position vector, J(q) is the Jacobian matrix, and the joint torque vector.

[0017] Based on the established dynamic model, the friction characteristics of the parallel robot under different conditions such as load, speed and temperature changes are identified, and the parameters to be identified are calculated and optimized through the dynamic equation to obtain the friction characteristics under actual working conditions.

[0018] Furthermore, in step 1 and step 2, in order to improve the control accuracy of the parallel robot in a complex environment, the present invention designs an enhanced iterative friction learning compensation controller. The controller optimizes and dynamically adjusts the friction compensation strategy by combining iterative learning and model reference adaptive control (MRAC) methods, thereby effectively reducing tracking errors.

[0019]

[0020] In the formula, y k (t) is the system output of the kth iteration, u k (t) is the control input of the kth iteration, F f,k (t) Friction compensation term, G system transfer function, v k is the speed of the parallel robot.

[0021] Specifically, the iterative learning module is used to analyze and update the friction compensation strategy. This module gradually improves the accuracy of the compensation strategy through a continuous iterative process. Each iteration adjusts the compensation strategy based on the latest feedback to ensure that the controller can adapt to environmental changes and system characteristics. The model reference adaptive control (MRAC) method is used to dynamically adjust the friction compensation strategy based on the resulting tracking error. This method adjusts the controller parameters to reduce the error by comparing the actual system response with the expected response of the reference model.

[0022] Furthermore, in step 2, an error function E is defined to calculate the Cartesian position tracking error. Given the target motion xd and the actual system state x, the error is Δx = xx d .

[0023] In order to achieve the above goals, an appropriate stiffness matrix K is selected, which can be adjusted according to the specific requirements of the task. K is used to describe the response of the system to position error; the Cartesian friction force h is calculated using the impedance-controlled stiffness matrix f,c , and the Cartesian friction force h is converted to f,c Project it into the joint space to calculate the joint space error matrix. In each iteration, collect data of Δx and J(q) and use this data to update E.

[0024] h f,c =-KΔx

[0025] E=J T (q)h f,c

[0026] Where K is the impedance-controlled stiffness matrix, E is the error matrix with the dimension [l, T], where l is the number of joints and T is the total number of iteration samples.

[0027] Furthermore, in step 2, the reward of the friction compensation strategy is defined as a global reward function P(k). A weight is assigned to the reward signal R(i,k) of each joint according to the importance of the joint in the multi-repetitive task. For key joints (the joints that are most important for task execution), a higher weight can be assigned.

[0028] The reward function is weighted and adjusted according to the size of the tracking error to better reflect the impact of the error on the control performance. The model reference adaptive control method (MRAC) is introduced to dynamically adjust the parameters of the reward function, thereby improving the control accuracy of the parallel robot in performing multiple repetitive tasks in complex environments.

[0029]

[0030] In the formula, w i is the weight of the ith element, and are the minimum and maximum values ​​of the enhanced signal, and γ is the adjustment coefficient.

[0031] Furthermore, in step 3, multiple repetitive tasks in complex environments that the parallel robot needs to perform are defined, such as polishing, assembly, disassembly, etc. These tasks require the robot to have high-precision force control capabilities during execution to ensure the quality and efficiency of the tasks.

[0032] In order to measure the contact force of the parallel robot end effector in real time, we installed a high-precision force sensor on the end effector of the parallel robot. This sensor can accurately measure the contact force and provide stable force feedback.

[0033] The force sensor is integrated into the parallel robot end effector or its appropriate position to ensure that the contact force data can be accurately captured. The integration position of the sensor is selected taking into account the actual requirements of the task and the measurement accuracy to maximize the performance and reliability of the sensor.

[0034] The present invention is equipped with a sensor data acquisition system to transmit the contact force data measured by the force sensor to the control system in real time. The system includes: a data acquisition module, a data transmission channel, and a data processing unit. The data acquisition system can realize the real-time transmission of the force sensor data, so that the control system can respond immediately and adjust the control strategy to adapt to the force changes in the actual task.

[0035] By receiving force sensor data in real time, the control system can obtain accurate feedback about the robot's interaction with the environment. This feedback is used to adjust the control strategy to improve the robustness of the force tracking controller. Combined with the force sensor data, the control system can dynamically adjust the controller parameters to optimize force tracking performance and reduce errors caused by environmental changes or uneven task loads.

[0036] Furthermore, in steps 2 and 3, for the global control of the force tracking controller, a reward function is defined: Apply reinforcement learning algorithms to dynamically adjust control gains, using reinforcement learning and optimization strategies to adapt to environmental changes. Then calculate the reinforcement signal in real time Update the control gain G according to the enhanced signal P , and adjust the learning rate and discount factor to optimize the convergence speed and effect of the force tracking strategy.

[0037]

[0038] G P =G P +G GP R GP

[0039] Furthermore, in step 4, the optimization of the overall control strategy includes a deep Q network (DQN) algorithm and a MRAC integrated control method.

[0040] In order to design the structure of the deep Q network (DQN), we first define the various layers of the network, including the input layer, multiple hidden layers, and the output layer. Then we implement the experience replay mechanism, store past experience in the replay pool, and randomly extract batches from it for training to break the correlation of the data.

[0041] To further stabilize the training process, we maintain an independent target network that works in conjunction with the main Q network. The weights of the target network are regularly updated to the weights of the main network, thereby preventing rapid fluctuations in the Q value and promoting smooth convergence of the training process.

[0042] The DQN algorithm continuously tests and corrects strategies through interaction with the environment. In each interaction, the parallel robot performs an action and obtains the corresponding reward and next state information. The algorithm updates the Q-value function based on these feedbacks, thereby gradually optimizing the strategy. DQN continuously adjusts the strategy parameters during the training process to find a strategy that can perform tasks efficiently. During the training process, we monitor the performance of the strategy and adjust the network structure and hyperparameters based on the training results to improve the learning effect of the strategy.

[0043] Furthermore, in step 4, the deep Q network (DQN) control strategy is combined with MRAC. Based on the control strategy generated by DQN, MRAC further adjusts the control strategy to adapt to the changing environment.

[0044] Specifically, DQN is responsible for learning high-level decision-making strategies in complex environments, and optimizing strategy selection to enable the system to achieve the overall task goal. Based on the DQN strategy, MRAC fine-tunes the controller parameters to cope with environmental changes and ensure the stability of system performance. Based on the actions provided by DQN, MRAC optimizes the accuracy of action execution by adjusting the controller gain in real time.

[0045] This fusion mechanism enables the high-level strategy of DQN and the low-level fine control of MRAC to complement each other to form a comprehensive control strategy. The coordination mechanism includes feedback on the error between the system output and the reference model output. MRAC adjusts the controller parameters based on this error, while DQN uses the information from these adjustments to update its strategy, thereby achieving closed-loop optimization.

[0046] Through the implementation of the above mechanisms, DQN provides a policy framework that adapts to the environment, while MRAC ensures the precise execution of the strategy in actual operations, and the coordination mechanism ensures the effective combination of the two, thereby enabling more efficient control of multiple repetitive tasks in complex environments.

[0047] In order to fully evaluate the adaptability and robustness of the parallel robot control strategy, we built a dynamically changing test environment. The environment includes movable obstacles, changing working conditions, uncertain environmental factors, etc. The design of this test environment is to fully simulate the various challenges that the robot may encounter in actual applications, so as to ensure the effectiveness and robustness of the control strategy under different conditions.

[0048] Advanced sensors and data acquisition equipment are installed on the parallel robot to record the robot's motion data, torque data and environmental changes in real time. A high-precision data acquisition system is configured to transmit sensor data to the analysis platform in real time. These data are used to monitor the robot's behavior and its interaction with the environment in detail, providing basic data support for the evaluation of control strategies.

[0049] Perform multiple repetitive tasks in a constructed dynamic test environment, including polishing, assembly, disassembly, etc. Record the detailed execution of each task under different environmental conditions, including: task completion time, control accuracy, force tracking error, etc.

[0050] By analyzing the above key performance indicators, the effectiveness and consistency of the control strategy under various environmental conditions are evaluated. These data are used to identify the advantages and disadvantages of the control strategy and provide a basis for further improvement. The optimized control strategy is then applied to the actual operating environment for verification. By applying it in real scenarios, the actual effect of the control strategy is evaluated to ensure its reliability and effectiveness in actual operation.

[0051] Finally, during the actual application process, system feedback data is collected, including system stability, adaptability, and task execution effect. These feedback data are used to evaluate the performance of the control strategy in the actual environment to optimize the control strategy. The control strategy is iteratively improved, and each round of iteration is improved on the basis of the previous round. Through continuous adjustment and optimization, the accuracy of the control strategy for the parallel robot to perform multiple repetitive tasks in complex environments, the efficiency of task execution, and the overall stability of the system are gradually improved.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] 1. The present invention proposes for the first time a reinforcement learning control method for a parallel robot with multiple repetitive tasks in a complex environment, which can significantly improve the control accuracy and adaptive ability of the robot in performing multiple repetitive tasks in a complex environment. Its characteristics and beneficial effects are:

[0054] 2. The iterative friction learning compensation controller of the present invention significantly improves the control accuracy by optimizing the friction compensation parameters and dynamically adjusting the control strategy, so that the robot can perform tasks more accurately. At the same time, the enhanced force tracking controller can accurately adjust the force output according to the real-time measurement data, which enhances the compliance of the robot.

[0055] 3. Through the fusion optimization of DQN and MRAC, the robot can efficiently perform multiple repetitive tasks in complex environments. DQN provides a high-level strategy to adapt to different environmental conditions, while MRAC ensures control accuracy and system stability, thereby improving the efficiency and reliability of task execution.

[0056] 4. The control system of the present invention can operate continuously and stably in the face of dynamic environments and changes in task loads. By dynamically adjusting the control strategy, the system can adapt to different operating conditions and environmental changes, improving the robustness and stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0058] Figure 1 This is a flow chart of the enhanced learning control method of the multi-repetitive task parallel robot in a complex environment of the present invention.

[0059] Figure 2 Schematic diagram of online and offline training process and module interaction for reinforcement learning.

[0060] Figure 3 This is a simulation scene diagram built on the simulation platform of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Of course, the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0062] Example 1

[0063] See also Figures 1 to 3The technical solution provided by this embodiment is that the present invention proposes an enhanced learning control method for a parallel robot with multiple repetitive tasks in a complex environment, so as to significantly improve the control accuracy and adaptability of the robot in performing multiple repetitive tasks in a complex environment. Firstly, a dynamic model of the UR10 parallel robot is established, which comprehensively considers the influencing factors such as joint friction, load and temperature change. Based on this model, the friction characteristics of the UR10 parallel robot under different working conditions are identified. Secondly, an enhanced iterative friction learning compensation controller is designed. The iterative learning module is used to analyze and update the friction compensation strategy. The model reference adaptive control method dynamically adjusts the friction compensation strategy according to the generated tracking error, thereby improving the control accuracy. In addition, an enhanced force tracking controller is designed to analyze the mechanical characteristics of the UR10 parallel robot in performing multiple repetitive tasks. The contact force is measured in real time through the force sensor module, and the corresponding strategy is adjusted with the help of the force tracking control module, thereby achieving accurate and compliant force tracking and compliance control. Based on the design of the above steps, the deep Q network and the model reference adaptive control method (MRAC) are integrated to realize the overall optimization control of the parallel robot, continuously optimize the execution efficiency of multiple repetitive tasks in complex environments, and dynamically adjust the control strategy according to different interactive tasks and environmental changes, thereby improving the system's adaptability and stability. Then, the system is compared with the sliding mode-adaptive-iterative learning control method of the repetitive processing of the parallel robot, showing that it has better performance in the face of more complex and dynamically changing environments.

[0064] Finally, a real complex environment with dynamic changes is built, and the monitoring function of the data acquisition and analysis module is configured to evaluate and optimize the enhanced learning control accuracy and adaptability of the multi-repetitive task parallel robot in a complex environment.

[0065] The method flow of this embodiment is as follows Figure 1 As shown, the specific method is as follows:

[0066] Step 1. Establish a dynamic model of the UR10 parallel robot that comprehensively considers factors such as joint friction, load, and temperature change:

[0067]

[0068] Where B(q) is the robot inertia matrix, is the Coriolis vector, g(q) is the gravity vector, is the joint friction vector, q is the joint position vector, J(q) is the Jacobian matrix, and τ is the joint torque vector. Based on the established dynamic model, the friction characteristics of the UR10 robot under different conditions such as load, speed and temperature changes are identified, and the parameters to be identified are calculated and optimized through the dynamic equations to obtain the friction characteristics under actual working conditions.

[0069] It is crucial to build a complex environment with dynamic changes and establish the dynamic model of the UR10 parallel robot in it. To improve the realism of modeling, obstacles should have various shapes and sizes, such as simulating mechanical equipment or objects of different forms. Obstacles of different shapes such as spheres, cubes, and cuboids are designed to test the adaptive ability of the UR10 parallel robot when facing different obstacle forms. And in the simulation, the friction coefficient at each joint of the robot is simulated to change with time or environmental factors (such as temperature). The friction coefficient is set to change dynamically, and by introducing noise functions or random variables, the friction characteristics show random fluctuations in the simulation. This setting can test the adaptive ability of the friction compensation controller when the friction changes.

[0070] In actual industrial scenarios, robots may need to switch quickly between different tasks. Therefore, a random task switching mechanism is set during the simulation process to allow the established UR10 parallel robot model to switch randomly between different tasks. Task types include object grasping, handling, and fine assembly. Each time the task is switched, the robot needs to readjust its actions and control strategies according to the requirements of the new task, which helps to fully test the adaptability and robustness of the control strategy.

[0071] Step 2. Design an enhanced iterative friction learning compensation controller. The iterative learning module is used to analyze and update the friction compensation strategy, and design a model reference adaptive control method according to the generated tracking error to dynamically adjust the friction compensation strategy, thereby improving the control accuracy.

[0072] Furthermore, in order to improve the control accuracy of the parallel robot in complex environments, the present invention designs an enhanced iterative friction learning compensation controller. The controller optimizes and dynamically adjusts the friction compensation strategy by combining iterative learning and model reference adaptive control (MRAC) methods, thereby effectively reducing tracking errors.

[0073]

[0074] In the formula, y k (t) is the system output of the kth iteration, u k (t) is the control input of the kth iteration, F f,k (t) Friction compensation term, G system transfer function, v k is the speed of the parallel robot.

[0075] Specifically, the iterative learning module is used to analyze and update the friction compensation strategy. This module gradually improves the accuracy of the compensation strategy through a continuous iterative process. Each iteration adjusts the compensation strategy based on the latest feedback to ensure that the controller can adapt to environmental changes and system characteristics. The model reference adaptive control (MRAC) method is used to dynamically adjust the friction compensation strategy based on the resulting tracking error. This method adjusts the controller parameters to reduce the error by comparing the actual system response with the expected response of the reference model.

[0076] Furthermore, in step 2, an error function E is defined to calculate the Cartesian position tracking error. Given the target motion x d and the actual system state x, the error is Δx = xx d .

[0077] In order to achieve the above goals, an appropriate stiffness matrix K is selected, which can be adjusted according to the specific requirements of the task. K is used to describe the response of the system to position error; the Cartesian friction force h is calculated using the impedance-controlled stiffness matrix f,c , and the Cartesian friction force h is converted to f,c Project it into the joint space to calculate the joint space error matrix. In each iteration, collect data of Δx and J(q) and use this data to update E.

[0078] h f,c =-KΔx

[0079] E=J T (q)h f,c

[0080] Where K is the impedance-controlled stiffness matrix, E is the error matrix with the dimension [l, T], where l is the number of joints and T is the total number of iteration samples.

[0081] Furthermore, the reward of the friction compensation strategy is defined as a global reward function P(k). The reward signal R(i,k) of each joint is weighted according to its importance in the multi-repetitive task. For key joints (the joints that are most important for task execution), higher weights can be assigned.

[0082] The reward function is weighted and adjusted according to the size of the tracking error to better reflect the impact of the error on the control performance. The model reference adaptive control method (MRAC) is introduced to dynamically adjust the parameters of the reward function, thereby improving the control accuracy of the UR10 robot in performing multiple repetitive tasks in complex environments.

[0083]

[0084] In the formula, w i is the weight of the ith joint, and are the minimum and maximum values ​​of the enhanced signal, and γ is the adjustment coefficient.

[0085] Step 3. Define the multiple repetitive tasks that the UR10 robot needs to perform in complex environments, such as polishing, assembly, disassembly, etc. These tasks require the robot to have high-precision force control capabilities during execution to ensure the quality and efficiency of the tasks.

[0086] In order to measure the contact force of the robot end effector in real time, we installed a high-precision force sensor on the end effector of the UR10 robot. The sensor can accurately measure the contact force and provide stable force feedback.

[0087] The force sensor is integrated into the robot end effector or its appropriate position to ensure that the contact force data can be accurately captured. The integration position of the sensor is selected taking into account the actual requirements of the task and the measurement accuracy to maximize the performance and reliability of the sensor.

[0088] The present invention is equipped with a sensor data acquisition system to transmit the contact force data measured by the force sensor to the control system in real time. The system includes: a data acquisition module, a data transmission channel, and a data processing unit. The data acquisition system can realize the real-time transmission of the force sensor data, so that the control system can respond immediately and adjust the control strategy to adapt to the force changes in the actual task.

[0089] By receiving force sensor data in real time, the control system can obtain accurate feedback about the robot's interaction with the environment. This feedback is used to adjust the control strategy to improve the robustness of the force tracking controller. Combined with the force sensor data, the control system can dynamically adjust the controller parameters to optimize force tracking performance and reduce errors caused by environmental changes or uneven task loads.

[0090] Furthermore, for the global control of the force tracking controller, the reward function is defined as Apply reinforcement learning algorithms to dynamically adjust control gains, using reinforcement learning and optimization strategies to adapt to environmental changes. Then calculate the reinforcement signal in real time Update the control gain G according to the enhanced signal P , and adjust the learning rate and discount factor to optimize the convergence speed and effect of the force tracking strategy.

[0091]

[0092] G P =G P +G GP R GP

[0093] The optimization of the overall control strategy includes the deep Q network (DQN) algorithm and MRAC integrated control method.

[0094] Step 4. To design the structure of the deep Q network (DQN), first define the various layers of the network, including the input layer, multiple hidden layers, and the output layer. Then implement the experience replay mechanism, store past experience in the replay pool, and randomly extract batches from it for training to break the correlation of the data.

[0095] To further stabilize the training process, we maintain an independent target network that works in conjunction with the main Q network. The weights of the target network are regularly updated to the weights of the main network, thereby preventing rapid fluctuations in the Q value and promoting smooth convergence of the training process.

[0096] The DQN algorithm continuously tests and corrects strategies through interaction with the environment. In each interaction, the UR10 parallel robot performs an action and obtains the corresponding reward and next state information. The algorithm updates the Q-value function based on these feedbacks, thereby gradually optimizing the strategy. DQN continuously adjusts the policy parameters during the training process to find a policy that can perform tasks efficiently. During the training process, we monitor the performance of the policy and adjust the network structure and hyperparameters based on the training results to improve the learning effect of the policy.

[0097] Furthermore, the deep Q network (DQN) control strategy is combined with MRAC. Based on the control strategy generated by DQN, MRAC further adjusts the control strategy to adapt to the changing environment.

[0098] Specifically, DQN is responsible for learning high-level decision-making strategies in complex environments, and optimizing strategy selection to enable the system to achieve the overall task goal. Based on the DQN strategy, MRAC fine-tunes the controller parameters to cope with environmental changes and ensure the stability of system performance. Based on the actions provided by DQN, MRAC optimizes the accuracy of action execution by adjusting the controller gain in real time.

[0099] This fusion mechanism enables the high-level strategy of DQN and the low-level fine control of MRAC to complement each other to form a comprehensive control strategy. The coordination mechanism includes feedback on the error between the system output and the reference model output. MRAC adjusts the controller parameters based on this error, while DQN uses the information from these adjustments to update its strategy, thereby achieving closed-loop optimization.

[0100] Through the implementation of the above mechanisms, DQN provides a policy framework that adapts to the environment, while MRAC ensures the precise execution of the strategy in actual operations, and the coordination mechanism ensures the effective combination of the two, thereby enabling more efficient control of multiple repetitive tasks in complex environments.

[0101] Step 5. To fully evaluate the adaptability and robustness of the UR10 robot control strategy, we built a dynamically changing test environment. The environment includes movable obstacles, changing working conditions, uncertain environmental factors, etc. The design of this test environment is to fully simulate the various challenges that the robot may encounter in actual applications, so as to ensure the effectiveness and robustness of the control strategy under different conditions.

[0102] Perform preset multiple repetitive tasks under different complex environmental conditions. The execution process of each task is monitored by sensors and detailed data is recorded. The core indicators of each task execution are recorded in detail, mainly including: the time required from the start to the completion of each task; the deviation of the robot from the target trajectory during the execution of the task, including position deviation and force tracking deviation; the error between the actual force and the expected force when the robot contacts the environment is recorded in real time during the simulation process to evaluate the performance of the force controller. By changing the working environment (such as loading different loads, adding obstacles or changing the friction coefficient), the control strategy performance of the UR10 robot under different task conditions is evaluated.

[0103] In the simulation, the effectiveness of the control strategy is evaluated in real time using the data analysis module. The performance of the control strategy in complex environments is comprehensively analyzed by comparing the task completion time, control accuracy, and force tracking error under different tasks and environmental conditions. Special attention is paid to the robustness of the robot's control strategy, that is, the stability and adaptability of the control strategy when the environment changes dynamically or uncertain factors appear. Through multiple task tests, it is analyzed whether the robot can maintain the expected performance in complex environments.

[0104] According to the feedback data collected during the simulation (such as torque data, control error, and execution time), the reinforcement learning controller and the model reference adaptive controller (MRAC) are adjusted. The control strategy is iteratively optimized using a feedback control loop, including: using the reward value fed back in the simulation to further optimize the control strategy by adjusting the Q-value update mechanism so that the robot can better adapt to complex tasks; dynamically adjusting the parameters in the MRAC according to the feedback tracking error to improve the system's ability to adapt to external changes such as friction and torque. Implement multiple rounds of iterations, each round of optimization based on the test data and feedback of the previous round. Gradually improve the execution efficiency, accuracy, and stability of the control strategy. In each round of iteration, the same multi-task test is repeatedly run on the simulation platform to evaluate the effect of the adjusted control strategy.

[0105] Verify on the actual UR10 robot hardware platform, build a complex test environment similar to the simulation, and introduce dynamically changing environmental factors (such as moving obstacles, load changes, etc.) to verify the performance of the control strategy in the simulation in the actual environment. Use real sensors and data acquisition equipment to collect motion data, torque data, etc. during task execution, and analyze whether the performance of the actual controller is consistent with the results obtained in the simulation. Through multiple task tests in actual operation, analyze the stability, adaptability and task execution efficiency of the control strategy, and compare them with the simulation results to ensure that they meet the expected performance standards.

[0106] The control strategy is further optimized through the feedback data in the experiment, especially the weights and parameters of the reinforcement learning algorithm (such as DQN) and the adaptive controller (MRAC) are adjusted for the uncertainty factors in the complex environment. Multiple rounds of iterative optimization are carried out in the experiment, and each iteration adjusts the algorithm based on the feedback data to improve the task execution efficiency and control accuracy. Finally, after several rounds of optimization and verification, it is ensured that the control strategy can achieve high-precision and high-robustness control effects in complex environments.

[0107] Example 2

[0108] This embodiment provides a comparison of the application of the enhanced learning control method (Example 1) designed by the present invention and the traditional adaptive control method for a multi-repetitive task parallel robot simulation experimental platform in a complex environment. The performance of the two algorithms in terms of convergence speed, stability, control accuracy, adaptability and flexibility was evaluated. The simulation experiment results show that the enhanced learning control method is superior to the traditional adaptive control in all the above evaluation aspects, especially in high-precision and dynamic complex environments. The adaptive control algorithm is crucial in complex environments with dynamic uncertainty. Traditional adaptive control methods have been widely used, but have limitations in terms of convergence speed and stability. This embodiment aims to compare the two methods through theoretical analysis and experimental verification.

[0109] The model reference adaptive control (MRAC) algorithm used in the reinforcement learning control framework designed by the present invention uses a known reference model to guide the system output so that it closely follows the output of the reference model. By adjusting the control parameters online, MRAC can quickly adapt to the dynamic changes of the system, thereby achieving rapid convergence and good stability. This method can effectively avoid excessive oscillation of the system when the parameters change, ensuring the stability of the control process. Traditional adaptive control relies on online estimation of system parameters and adjusts the parameters of the controller based on the estimation results. Although this method is effective in some cases, its adaptation process is usually slow, especially in a rapidly changing environment, the performance often decreases significantly. Due to its high dependence on algorithm design and external disturbances, traditional adaptive control may face the risk of instability when responding to dynamic changes.

[0110] The MRAC-integrated reinforcement learning control method achieves fast convergence by closely following the reference model, and can maintain stability when system parameters change, avoiding excessive oscillation. This makes MRAC perform well in dealing with dynamic environments. In contrast, traditional adaptive control converges slowly, especially when parameters change dramatically, the system may become unstable, and it is heavily dependent on the specific algorithm design and the influence of the external environment.

[0111] In terms of control accuracy, the enhanced learning control method integrated with MRAC shows significant advantages. This method can accurately track the reference model and significantly reduce the tracking error. It is suitable for application scenarios with high precision requirements, such as parallel robot control and flight control. Relatively speaking, the control accuracy of traditional adaptive control is low, and large steady-state errors and overshoot may occur, especially in complex dynamic environments, and the performance is limited by the accuracy of its parameter estimation.

[0112] The MRAC-integrated reinforcement learning control method has strong adaptability and can quickly respond to environmental changes and fluctuations in system parameters. This feature makes it widely used in various dynamic systems, especially in uncertain environments. In contrast, traditional adaptive control has relatively weak adaptability and long response time, making it difficult to effectively cope with rapidly changing environments. It usually requires frequent readjustment of algorithm parameters to maintain system performance.

[0113] In summary, the MRAC-integrated reinforcement learning control method is superior to the traditional adaptive control in terms of convergence speed, stability, control accuracy and adaptability. The advantages of the MRAC-integrated reinforcement learning control method make it an ideal choice for adaptive control in high-precision and dynamic environments. Future research can further explore the performance optimization and practical application of the MRAC-integrated reinforcement learning control method in more complex application scenarios. Table 1 shows the experimental data of the MRAC-integrated reinforcement learning control method and traditional adaptive control under the same conditions.

[0114] Table 1 Experimental data of the enhanced learning control method integrating MRAC and traditional adaptive control under the same conditions

[0115]

[0116] The experiment used a simulation test platform. The controller allowed torque control of the manipulator and recorded the required data (robot joint and Cartesian positions, velocities, robot Jacobian matrix). The control / sampling frequency was set to 125 Hz, and the equipped force sensor allowed recording of the interaction force / torque.

[0117] A free motion task along the Cartesian direction x is chosen as the target task. Friction learning has to be performed on the manipulator without external forces / torques acting on it. Once the learning process is complete, friction can be compensated in the interactive task, thus improving the controller performance. The learning method is applied to an automotive industry assembly task.

[0118] Table 2 shows the experimental data. The results show that the reinforcement learning control method integrated with MRAC performs well in friction compensation and force tracking, and can converge quickly and maintain high accuracy.

[0119] Table 2 Experimental data of the reinforcement learning control method integrating MRAC in terms of friction compensation and force tracking

[0120] Experimental indicators MRAC Traditional adaptive control Friction compensation error(N) 0.05 0.2 Force tracking error(N) 0.1 0.5 Convergence time (s) 2.0 4.5 Steady-state error (N) 0.05 0.3

[0121] Comparative studies of simulation experiments show that the MRAC-integrated reinforcement learning control method is superior to traditional adaptive control in terms of convergence, stability, control accuracy and adaptability. The MRAC-integrated reinforcement learning control method is more suitable for high-precision and dynamic environments. Future research will focus on further optimizing the performance of the MRAC-integrated reinforcement learning control method and exploring its application in more complex scenarios.

[0122] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A reinforcement learning control method for a multi-task parallel robot in a complex environment, characterized in that: The following steps are involved: Step 1: Establish a dynamic model of the parallel robot that comprehensively considers the influencing factors of joint friction, load and temperature change, and identify the friction characteristics of the parallel robot under different working conditions based on the model; Step 2: Design an enhanced iterative friction learning compensation controller. The iterative learning module is used to analyze and update the friction compensation strategy, and dynamically adjust the friction compensation strategy based on the tracking error generated by the model reference adaptive control method. Step 3: Design an enhanced force tracking controller, analyze the mechanical characteristics of the parallel robot in performing multiple repetitive tasks, measure the contact force in real time through the force sensor module, and adjust the corresponding strategy with the help of the force tracking control module; Step 4: Based on steps 2) and 3), the deep Q network and the model reference adaptive control method MRAC are integrated to achieve the overall optimization control of the parallel robot, continuously optimize the execution efficiency of multiple repetitive tasks in complex environments, dynamically adjust the control strategy according to different interactive tasks and environmental changes, and improve the system's adaptability and stability; Step 5: Build a real complex environment with dynamic changes and configure the monitoring function of the data acquisition and analysis module to evaluate and optimize the enhanced learning control accuracy and adaptability of the multi-task parallel robot in a complex environment.

2. The enhanced learning control method for a multi-task parallel robot in a complex environment according to claim 1, characterized in that: In step 1, a dynamic model of the parallel robot is established which comprehensively considers the influencing factors of joint friction, load and temperature change: In the formula, is the robot inertia matrix, is the Coriolis vector, g(q) is the gravity vector, is the joint friction vector, q is the joint position vector, J(q) is the Jacobian matrix, and τ is the joint torque vector. Based on the established dynamic model, the friction characteristics of the parallel robot under different conditions of load, speed and temperature changes are identified, and the parameters to be identified are calculated and optimized through the dynamic equations to obtain the friction characteristics under actual working conditions.

3. The reinforcement learning control method for a multi-task parallel robot in a complex environment according to claim 1, characterized in that: In step 1 and step 2, the process of designing the reinforced iterative friction learning compensator is modeled as an iterative optimization process based on reinforced tracking learning: In the formula, y k (t) is the system output of the kth iteration, u k (t) is the control input of the kth iteration, F f,k (t) is the friction compensation term, G is the system transfer function, v k is the speed of the parallel robot.

4. The method for enhanced learning control of a parallel robot with multiple repetitive tasks in a complex environment according to claim 3, characterized in that: In step 2, the error function E is defined to calculate the Cartesian position tracking error. Given the target motion x d and the actual system state x, the error is Δx = xx d , select an appropriate stiffness matrix K, which can be adjusted according to the specific requirements of the task, and use K to describe the system's response to position error; use the impedance-controlled stiffness matrix to calculate the Cartesian friction force h f,c , and the Cartesian friction force h is converted to f,c Project it into the joint space to calculate the joint space error matrix. In each iteration, collect the data of Δx and J(q) and use them to update E. h f,c =-KΔx E=J T (q)h f,c Where K is the impedance-controlled stiffness matrix, E is the error matrix with the dimension [l, T], where l is the number of joints and T is the total number of iteration samples.

5. The method for enhanced learning control of a parallel robot with multiple repetitive tasks in a complex environment according to claim 4, characterized in that: In step 2, the reward of the friction compensation strategy is defined as a global reward function P(k). According to the importance of the joint in the multi-repetitive task, a weight is assigned to the reward signal R(i, k) of each joint. For the key joint, a higher weight is assigned. The reward function is weighted and adjusted according to the size of the tracking error. The model reference adaptive control method MRAC is introduced to dynamically adjust the parameters of the reward function. In the formula, w i is the weight of the ith joint, and are the minimum and maximum values ​​of the enhanced signal, and γ is the adjustment coefficient.

6. The method for enhanced learning control of a parallel robot with multiple repetitive tasks in a complex environment according to claim 4, characterized in that: In step 3, multiple repetitive tasks in complex environments that the parallel robot needs to perform are defined, a high-precision force sensor is installed on the end effector of the parallel robot to measure the contact force in real time, and the force sensor is then integrated into the robot end effector or an appropriate position to obtain accurate contact force data. Then, a sensor data acquisition system is configured to collect the measured contact force f s Real-time transmission to the control system ensures that the force sensor provides stable measurement data during multiple repetitive tasks.

7. The method for enhanced learning control of a parallel robot with multiple repetitive tasks in a complex environment according to claim 6, characterized in that: In step 2 and step 3, for the global control of the force tracking controller, define the reward function Apply reinforcement learning algorithms to dynamically adjust control gains, use reinforcement learning and optimization strategies to adapt to environmental changes, and then calculate reinforcement signals in real time Update the control gain G according to the enhanced signal P , and adjust the learning rate and discount factor to optimize the convergence speed and effect of the force tracking strategy; At the initial control gain G P and G d In this paper, we perform tasks and record real-time data from force sensors, analyze the mechanical response characteristics of the parallel robot in these tasks, monitor force changes in real time and adjust control strategies, cope with dynamic changes in complex environments, perform multiple repetitive tasks in complex environments, and evaluate the performance of force tracking controllers in these tasks.

8. The method for enhanced learning control of a parallel robot with multiple repetitive tasks in a complex environment according to claim 7, characterized in that: In step 4, the overall control strategy is optimized, including the deep Q network DQN algorithm and the MRAC integrated control method. In order to design the structure of the deep Q network DQN, the various layers of the network are first defined, including the input layer, multiple hidden layers and the output layer, and then the experience replay mechanism is implemented to store past experience in the replay pool, from which batches are randomly selected for training to break the correlation of the data.

9. The method for enhanced learning control of a parallel robot with multiple repetitive tasks in a complex environment according to claim 8, characterized in that: In step 4, the deep Q network DQN control strategy and MRAC are combined. Based on the control strategy generated by DQN, MRAC further adjusts the control strategy. DQN is responsible for learning high-level decision-making strategies in complex environments and optimizing strategy selection to enable the system to achieve the goal of the overall task. MRAC fine-tunes the controller parameters based on the DQN strategy. Based on the actions provided by DQN, MRAC optimizes the accuracy of action execution by adjusting the controller gain in real time.

10. The method for enhanced learning control of a parallel robot with multiple repetitive tasks in a complex environment according to claim 9, characterized in that: In the steps 4 and 5, a dynamically changing test environment is constructed, including movable obstacles, changing working conditions and uncertain environmental factors. Advanced sensors and data acquisition equipment are installed on the parallel robot to record the robot's motion data, torque data and environmental changes in real time.

Images