Mechanical arm reinforcement learning control platform based on digital twin drive and construction method

Abstract

The invention discloses a mechanical arm reinforcement learning control platform based on digital twinning driving and a construction method. The platform is provided with a virtual control real module, a real control virtual module and a training optimization module. Firstly, a physical mechanical arm is controlled through a virtual control real function and a digital physical model, and real-time control is achieved; and the real-control virtual function updates the joint angle, speed and other states of the digital entity model in real time through sensing data feedback of the entity mechanical arm, so that the virtual model and the physical model are kept synchronous. In addition, the platform carries out reinforcement learning training in a digital entity model through an optimization training module, the trained model can be put into the entity model for testing, a complete optimization training strategy is finally formed, and the operation efficiency and precision are improved. The platform supports 3D simulation and dynamic data updating, high-fidelity digital twinning of the mechanical arm is achieved, the platform can be widely applied to the fields of industrial automation, remote control, intelligent manufacturing and the like, the control efficiency can be improved, the operation process can be optimized, and the safety and stability of mechanical arm operation can be improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent manufacturing and robotics technology, and in particular to a robotic arm reinforcement learning control platform and a construction method based on digital twin drive. Background Art

[0002] In the field of modern industry and intelligent manufacturing, robotic arms are widely used in various complex automation tasks, such as welding, assembly, handling, etc. However, traditional robotic arm control methods usually rely on pre-programmed paths and fixed control strategies, which cannot flexibly respond to real-time changing environments and operational requirements. At the same time, robotic arms may encounter various uncertainties in actual applications, such as sensor errors, mechanical deviations, and external disturbances, which can easily affect the control accuracy and operation stability of the robotic arm.

[0003] As a cutting-edge technology that combines virtuality and reality, digital twin technology has attracted widespread attention in the manufacturing industry in recent years. By building a digital robotic arm model that is consistent with the physical equipment, the digital twin can reflect the state of the physical system in real time, thereby realizing real-time monitoring, simulation and prediction of the physical equipment. Especially in the field of robotic arm control, the control platform based on digital twin can accurately simulate the motion state of the robotic arm through the digital robotic arm model, further improving the control accuracy and response speed.

[0004] Existing digital twin systems usually implement one-way interaction from virtual to physical (virtual controls the physical) or physical to virtual (physical controls the virtual), but these systems have some shortcomings in actual operation. For example, the implementation of virtual control of the physical requires complex communication and control algorithms, while physical control of the virtual requires efficient data transmission and update mechanisms to ensure that the digital robotic arm model is highly consistent with the physical model. In addition, in terms of operation optimization, traditional methods lack effective training methods, which makes it difficult to dynamically adjust the control strategy according to the actual environment, reducing the working efficiency of the robotic arm. Summary of the invention

[0005] In view of the above-mentioned technical deficiencies, the purpose of the present invention is to provide a digital twin-driven robotic arm reinforcement learning control platform and construction method, which combines the three functional modules of virtual control of real, real control of virtual and training optimization, aiming to build an efficient, real-time and accurate robotic arm control system. The virtual control of the real module realizes remote control of the physical robotic arm, and the real control of the virtual module realizes real-time feedback and synchronous update of the physical robotic arm state. At the same time, with the help of reinforcement learning to optimize the control strategy, the operating efficiency and accuracy of the robotic arm are improved. This innovative control platform can be widely used in industrial automation, remote control and intelligent manufacturing, effectively solving the limitations of traditional control methods.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] The present invention provides a mechanical arm reinforcement learning control platform based on digital twin drive and a construction method thereof, wherein the control platform comprises a virtual control real module, a real control virtual module and a training optimization module; the virtual control real module realizes remote real-time control of a physical mechanical arm through a digital mechanical arm model; the real control virtual module is used to realize real-time feedback of the state of the physical mechanical arm, by collecting real-time data of various sensors arranged on the mechanical arm, including joint angle, speed, and position information, and synchronously transmitting these data to the digital mechanical arm model to ensure that the motion state of the digital mechanical arm model is completely consistent with that of the physical mechanical arm; the training optimization module automatically generates an optimized control strategy by performing reinforcement learning training in the digital mechanical arm model, and applies it to the physical mechanical arm; the construction method of the control platform comprises the following steps:

[0008] Step 1. Import the URDF model file and introduce the robotic arm model into Unity through URDF Importer to form a digital robotic arm model;

[0009] Step 2: Build a digital environment in Unity that matches the actual environment;

[0010] Step 3. Configure the communication environment with ROS in Unity;

[0011] Step 4: Build a virtual control real module, specifically: monitor the / joint_states topic of the physical robot arm, synchronize the joint state to the Unity model, and realize the physical control of the digital robot arm;

[0012] Step 5: Build a real-control virtual module, specifically: the digital robotic arm model sends the joint angle and the desired end position to ROS MoveIt, obtains the trajectory and sends it to the digital robotic arm model, and controls the physical robotic arm to execute, realizing virtual control of the physical object;

[0013] Step 6: Build a training optimization module, specifically: use ML-Agents for reinforcement learning training and optimize the robotic arm control strategy.

[0014] Preferably, step 1 specifically includes the following contents:

[0015] S11. Install the URDF Importer plugin in Unity’s Package Manager.

[0016] S12. Import the URDF file and related resource files (such as mesh files in STL or DAE format) of the ROCR6 robot arm into the Assets folder of Unity.

[0017] S13. Select URDF Importer>Import URDF in the Unity menu to load the ROCR6 robotic arm model into the Unity scene.

[0018] Preferably, step 2 specifically includes the following contents:

[0019] S21. Make precise measurements based on the physical layout and dimensions of the actual environment and create a 3D model that is consistent with the real environment;

[0020] S22. Set the physical parameters of objects in the Unity scene, including friction coefficient, collision volume, mass, and gravity, to ensure that the behavior of virtual objects is consistent with real objects, thereby achieving high-fidelity physical simulation effects.

[0021] Preferably, step 3 specifically includes the following contents:

[0022] S31. Import the two components ros-tcp-connector and unityroboticsvisualizations through Window / Package Manager on the Unity platform;

[0023] S32. After S31 is executed, Unity will have an additional option Robotics, configure ROS IP and ROS Port;

[0024] S33. Compile ROS-TCP-Endpoint into the ROS workspace.

[0025] Preferably, step 4 specifically includes the following contents:

[0026] S41: According to step 3, configure the ROS communication plug-in in Unity, subscribe to the / joint_states topic, and obtain real-time joint state data from the physical robotic arm;

[0027] S42: monitor the / joint_states topic to receive joint angle information, and parse the information into specific values ​​for each joint;

[0028] S43: Mapping the parsed joint angle values ​​to the digital robotic arm model in Unity, and converting the joint state data into angle information of each joint in Unity to maintain data consistency;

[0029] S44: Use smooth interpolation algorithm to optimize the movement of Unity robot arm, reduce the jitter caused by data transmission delay or update frequency difference, and ensure the smoothness and real-time movement of digital robot arm model.

[0030] Preferably, step 5 specifically includes the following contents:

[0031] S51: Create and start a ROS service node in Unity, so that it can send the joint angles and expected end pose of the digital robot model to the ROS end. The expected end pose will use the position of the ball in Unity as the target;

[0032] S52: Send the current joint angle of the digital robot arm and the position of the ball to the ROS end through a ROS service request;

[0033] S53: monitor and receive the service request sent by the Unity end, extract the joint angle and the expected end position from the request, and process it in MoveIt;

[0034] S54: Use MoveIt to perform path planning on the received joint angles and target postures, and calculate a trajectory that enables the robot arm to reach the target posture from the current position; the trajectory includes the joint angles and posture changes at each time step;

[0035] S55: Send the calculated trajectory as the response data of the ROS service back to the Unity end, so that the digital robot arm model in Unity can perform corresponding movements according to the planned trajectory;

[0036] S56: receiving the trajectory data returned by ROS in Unity, parsing the joint angles of each time step, and applying them to the joints of the Unity digital robotic arm model, so that the digital robotic arm model moves along the trajectory planned by MoveIt in the digital environment;

[0037] S57: Apply the same trajectory to the physical robot arm, so that it performs the same movements as the Unity digital robot arm model, achieving synchronous movement of virtual control of the physical object.

[0038] Preferably, step 6 specifically includes the following contents:

[0039] S61: Install and configure the ML-Agents plug-in in Unity to create a training environment for the control of the digital robotic arm model, including setting the initial position and target position of the robotic arm, as well as the reward and penalty rules;

[0040] S62: defining the joint angle, end position and target position information of the robot arm as an observation space, so that the digital robot arm model can perceive the current state and target of the robot arm;

[0041] S63: setting the action space of the Agent, including the control input of each joint, including the angular velocity or angle adjustment, so that the digital robot arm model outputs an action according to the strategy at each time step to adjust the posture of the robot arm toward the target;

[0042] S64: Define the reward function, design rewards and penalties based on the distance between the end of the robotic arm and the target position, whether the target is successfully reached, and action efficiency factors, and encourage the digital robotic arm model to approach the target in an efficient and stable manner.

[0043] S65: Run ML-Agents training, use reinforcement learning algorithms (such as PPO or SAC) to train the control strategy of the robotic arm, and continuously optimize the performance of the agent in different scenarios.

[0044] S66: Observe and debug the learning effect of the agent during the training process, and gradually improve the control accuracy and stability of the robotic arm by adjusting hyperparameters, reward functions, or adding new constraints.

[0045] S67: After the training is completed, the trained model is deployed in Unity to verify its effect in the real simulation environment to ensure that it can complete the target action stably and accurately.

[0046] S68: If necessary, export the optimized model in ONNX format and deploy it on the ROS side to apply it to the physical robot arm, so that it can use the optimized control strategy in actual operation to achieve more efficient and precise motion control.

[0047] The beneficial effects of the present invention are:

[0048] (1) Synchronous update of virtual and real: Through the real control of the virtual module, the platform can collect the joint angle, speed and other status information of the physical robot arm in real time, and synchronously update it to the digital twin model, that is, the digital robot arm model, to achieve dynamic consistency between the virtual model and the physical model. This real-time feedback mechanism enables users to accurately monitor the operating status of the robot arm, helps to quickly detect and troubleshoot potential faults, and improves the reliability of the system.

[0049] (2) Optimizing training efficiency: The training optimization module uses reinforcement learning algorithms to perform simulation training on the digital twin model, and the generated optimized control strategy can be directly applied to the physical robot arm. Through a large number of training iterations in the virtual environment, the system can automatically optimize the control strategy and improve the operational efficiency and adaptability of the robot arm in complex tasks. In this way, the optimized control strategy can achieve efficient and accurate action execution in actual operation, shortening the time cost of strategy adjustment.

[0050] (3) High-fidelity 3D simulation and dynamic data update: The platform is built on Unity and has high-fidelity 3D simulation and real-time data update capabilities. This function ensures that the user's operating experience of the robotic arm in the virtual environment is consistent with the actual operation, providing operators with intuitive and realistic control feedback, further improving operational safety and system adaptability.

[0051] (4) Wide applicability and reliability: The platform is applicable to industrial automation, intelligent manufacturing, remote control and other fields. It improves the safety, stability and operational efficiency of robot control through efficient digital twin technology. The system has strong robustness and adaptability in complex environments, which can reduce the risks in the operation of the robot, improve production efficiency and optimize the operation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 is a flow chart of a method for constructing a digital twin robotic arm control platform based on Unity provided in this embodiment;

[0054] Figure 2 It is the robot arm model used in the specific implementation case;

[0055] Figure 3 It is the interaction between gazebo (virtual objects) and Unity virtual reality;

[0056] Figure 4 It is the interactive tracking of real objects and Unity virtual reality; DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] like Figure 1 to Figure 2 As shown, this embodiment provides a robot arm reinforcement learning control platform and construction method based on digital twin drive.

[0059] The platform includes three functional modules: virtual control of the real, real control of the virtual, and training optimization. The platform takes the digital twin model as the core and combines the virtual-real interactive control method to achieve control of the robotic arm, real-time data feedback, and intelligent strategy optimization, thus improving the control accuracy, operation efficiency, and adaptability of the robotic arm in practical applications.

[0060] Firstly, the virtual control real module realizes remote real-time control of the physical robotic arm through the digital robotic arm model;

[0061] Secondly, the real-control virtual module is used to realize real-time feedback on the state of the physical robotic arm. By collecting real-time data from various sensors arranged on the robotic arm, including joint angles, speeds, and position information, these data are synchronously transmitted to the digital robotic arm model to ensure that the motion state of the digital robotic arm model is completely consistent with that of the physical robotic arm. Arranging corresponding sensors to obtain corresponding data belongs to the prior art and will not be described in detail.

[0062] Finally, the training optimization module automatically generates an optimized control strategy by performing reinforcement learning training in the digital robotic arm model and applies it to the physical robotic arm.

[0063] During the training process, the platform simulates different working environments and conditions, conducts multiple rounds of training and optimization on the control strategy, and improves its adaptability and execution efficiency in complex tasks. After the training is completed, the optimized control strategy is deployed to the physical robot arm, and after actual verification, it ensures that more efficient and accurate control effects can be achieved in actual applications.

[0064] The platform is built on Unity and has high-precision 3D simulation functions and dynamic data update capabilities, and is widely used in industrial automation, remote control, intelligent manufacturing and other fields. The present invention not only significantly improves the control efficiency and operational stability of the robotic arm, but also optimizes the control strategy through reinforcement learning, effectively improving the adaptability and robustness of the system, and can achieve safe and efficient robotic arm operations in complex environments.

[0065] The control platform construction method comprises the following steps:

[0066] Step 1: Import the URDF model file and use the URDF Importer to introduce the robotic arm model into Unity to form a digital robotic arm model. Specifically:

[0067] S11. Install the URDF Importer plugin in Unity’s Package Manager.

[0068] S12. Import the URDF file and related resource files (such as mesh files in STL or DAE format) of the ROCR6 robot arm into the Assets folder of Unity.

[0069] S13. Select URDF Importer>Import URDF in the Unity menu to load the ROCR6 robotic arm model into the Unity scene.

[0070] Step 2: Build a digital environment in Unity that matches the actual environment;

[0071] S21. Take precise measurements based on the physical layout and dimensions of the actual environment and create a 3D model that is consistent with the real environment.

[0072] S22. Set the physical parameters of objects in the Unity scene, including friction coefficient, collision volume, mass, and gravity, to ensure that the behavior of virtual objects is consistent with real objects, thereby achieving high-fidelity physical simulation effects.

[0073] Step 3. Configure the communication environment with ROS in Unity;

[0074] S31. Import the two components ros-tcp-connector and unity robotics visualizations through Window / Package Manager on the Unity platform;

[0075] S32. After S31 is executed, Unity will have an additional option Robotics, configure ROS IP and ROS Port;

[0076] S33. Compile ROS-TCP-Endpoint into the ROS workspace;

[0077] Before building a real physical platform, it is recommended to use Gazebo as a virtual physical robot arm for testing, and then transplant it to the physical platform.

[0078] Step 4: Build a virtual control real module, specifically: monitor the / joint_states topic of the physical robot arm, synchronize the joint state to the Unity model, and realize the physical control of the digital robot arm;

[0079] S41: According to step 3, configure the ROS communication plug-in in Unity, subscribe to the / joint_states topic, and obtain real-time joint state data from the physical robotic arm;

[0080] S42: monitor the / joint_states topic to receive joint angle information, and parse the information into specific values ​​for each joint;

[0081] S43: Mapping the parsed joint angle values ​​to the digital robotic arm model in Unity, and converting the joint state data into angle information of each joint in Unity to maintain data consistency;

[0082] S44: Use smooth interpolation algorithm to optimize the movement of Unity robot arm, reduce the jitter caused by data transmission delay or update frequency difference, and ensure the smoothness and real-time movement of digital robot arm model.

[0083] Step 5: Build a real-control virtual module, specifically: the digital robotic arm model sends the joint angle and the desired end position to ROS MoveIt, obtains the trajectory and sends it to the digital robotic arm model, and controls the physical robotic arm to execute, realizing virtual control of the physical object;

[0084] S51: Create and start a ROS service node in Unity, so that it can send the joint angles and expected end pose of the digital robot model to the ROS end. The expected end pose will use the position of the ball in Unity as the target;

[0085] S52: Pack the current joint angle of the digital robot and the position of the ball (as the expected end position) and send them to the ROS end through a ROS service request;

[0086] S53: monitor and receive the service request sent by the Unity end, extract the joint angle and the expected end position from the request, and process it in MoveIt;

[0087] S54: Use MoveIt to perform path planning on the received joint angles and target postures, and calculate a trajectory that enables the robot arm to reach the target posture from the current position; the trajectory includes the joint angles and posture changes at each time step;

[0088] S55: Send the calculated trajectory as the response data of the ROS service back to the Unity end, so that the digital robot arm model in Unity can perform corresponding movements according to the planned trajectory;

[0089] S56: receiving the trajectory data returned by ROS in Unity, parsing the joint angles of each time step, and applying them to the joints of the Unity digital robotic arm model, so that the digital robotic arm model moves along the trajectory planned by MoveIt in the digital environment;

[0090] S57: Apply the same trajectory to the physical robot arm, so that it performs the same movements as the Unity digital robot arm model, achieving synchronous movement of virtual control of the physical object.

[0091] Through steps 4 and 5, the virtual-to-real and real-to-virtual functions can be realized. The interactive tracking between virtual and real objects using gazebo and Unity is as follows: Figure 3As shown in the figure, the joint movement range is -180°~180°, and the maximum error is within 2°, and the tracking effect is good. After completing steps 4 and 5, you can deploy it on a physical robot arm for testing. The interactive tracking between the real object and Unity virtual reality is as follows: Figure 4 As shown, the maximum error is still within 2°.

[0092] Step 6: Build a training optimization module, specifically: use ML-Agents for reinforcement learning training and optimize the robotic arm control strategy.

[0093] S61: Install and configure the ML-Agents plug-in in Unity to create a training environment for the control of the digital robotic arm model, including setting the initial position and target position of the robotic arm, as well as the reward and penalty rules;

[0094] S62: defining the joint angle, end position and target position information of the robot arm as an observation space, so that the digital robot arm model can perceive the current state and target of the robot arm;

[0095] S63: setting the action space of the Agent, including the control input of each joint, including the angular velocity or angle adjustment, so that the digital robot arm model outputs an action according to the strategy at each time step to adjust the posture of the robot arm toward the target;

[0096] S64: Define the reward function, design rewards and penalties based on the distance between the end of the robotic arm and the target position, whether the target is successfully reached, and action efficiency factors, and encourage the digital robotic arm model to approach the target in an efficient and stable manner.

[0097] S65: Run ML-Agents training, use reinforcement learning algorithms (such as PPO or SAC) to train the control strategy of the robotic arm, and continuously optimize the performance of the agent in different scenarios.

[0098] S66: Observe and debug the learning effect of the agent during the training process, and gradually improve the control accuracy and stability of the robotic arm by adjusting hyperparameters, reward functions, or adding new constraints.

[0099] S67: After the training is completed, the trained model is deployed in Unity to verify its effect in the real simulation environment to ensure that it can complete the target action stably and accurately.

[0100] S68: If necessary, export the optimized model in ONNX format and deploy it on the ROS side to apply it to the physical robot arm, so that it can use the optimized control strategy in actual operation to achieve more efficient and precise motion control.

[0101] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A digital twin-driven robotic arm reinforcement learning control platform and construction method, characterized in that: The control platform includes a virtual control real module, a real control virtual module and a training optimization module; the virtual control real module realizes remote real-time control of the physical robotic arm through the digital robotic arm model; the real control virtual module is used to realize real-time feedback of the state of the physical robotic arm, by collecting real-time data of various sensors arranged on the robotic arm, including joint angle, speed, and position information, and synchronously transmitting these data to the digital robotic arm model to ensure that the motion state of the digital robotic arm model is completely consistent with that of the physical robotic arm; The training optimization module automatically generates an optimized control strategy by performing reinforcement learning training in a digital robotic arm model and applies it to a physical robotic arm; the construction method of the control platform includes the following steps: Step 1. Import the URDF model file and introduce the robotic arm model into Unity through URDF Importer to form a digital robotic arm model; Step 2: Build a digital environment in Unity that matches the actual environment; Step 3. Configure the communication environment with ROS in Unity; Step 4: Build a virtual control real module, specifically: monitor the / joint_states topic of the physical robot arm, synchronize the joint state to the Unity model, and realize the physical control of the digital robot arm; Step 5: Build a real-control virtual module, specifically: the digital robotic arm model sends the joint angle and the desired end position to ROS MoveIt, obtains the trajectory and sends it to the digital robotic arm model, and controls the physical robotic arm to execute, realizing virtual control of the physical object; Step 6: Build a training optimization module, specifically: use ML-Agents for reinforcement learning training and optimize the robotic arm control strategy.

2. The digital twin-driven robotic arm reinforcement learning control platform and construction method according to claim 1 is characterized in that: Step 2 specifically includes the following: S21. Make precise measurements based on the physical layout and dimensions of the actual environment and create a 3D model that is consistent with the real environment; S22. Set the physical parameters of objects in the Unity scene, including friction coefficient, collision volume, mass, and gravity, to ensure that the behavior of virtual objects is consistent with real objects, thereby achieving high-fidelity physical simulation effects.

3. The digital twin-driven robotic arm reinforcement learning control platform and construction method according to claim 2 is characterized in that: Step 3 specifically includes the following: S31. Import the two components ros-tcp-connector and unityroboticsvisualizations through Window / Package Manager on the Unity platform; S32. After S31 is executed, Unity will have an additional option Robotics, configure ROS IP and ROS Port; S33. Compile ROS-TCP-Endpoint into the ROS workspace.

4. The digital twin-driven robotic arm reinforcement learning control platform and construction method according to claim 3 is characterized in that: Step 4 specifically includes the following: S41: According to step 3, configure the ROS communication plug-in in Unity, subscribe to the / joint_states topic, and obtain real-time joint state data from the physical robotic arm; S42: monitor the / joint_states topic to receive joint angle information, and parse the information into specific values ​​for each joint; S43: Mapping the parsed joint angle values ​​to the digital robotic arm model in Unity, and converting the joint state data into angle information of each joint in Unity to maintain data consistency; S44: Use smooth interpolation algorithm to optimize the movement of Unity robot arm, reduce the jitter caused by data transmission delay or update frequency difference, and ensure the smoothness and real-time movement of digital robot arm model.

5. The digital twin-driven robotic arm reinforcement learning control platform and construction method according to claim 4, characterized in that: Step 5 specifically includes the following: S51: Create and start a ROS service node in Unity, so that it can send the joint angles and expected end pose of the digital robot model to the ROS end. The expected end pose will use the position of the ball in Unity as the target; S52: Send the current joint angle of the digital robot arm and the position of the ball to the ROS end through a ROS service request; S53: monitor and receive the service request sent by Unity, extract the joint angle and expected end position from the request, and process it in MoveIt; S54: Use MoveIt to perform path planning on the received joint angles and target postures, and calculate a trajectory that enables the robot arm to reach the target posture from the current position; the trajectory includes the joint angles and posture changes at each time step; S55: Send the calculated trajectory as the response data of the ROS service back to the Unity end, so that the digital robot arm model in Unity can perform corresponding movements according to the planned trajectory; S56: receiving the trajectory data returned by ROS in Unity, parsing the joint angles of each time step, and applying them to the joints of the Unity digital robotic arm model, so that the digital robotic arm model moves along the trajectory planned by MoveIt in the digital environment; S57: Apply the same trajectory to the physical robotic arm, so that it performs the same movements as the Unity digital robotic arm model, achieving synchronous actions of virtual control of the physical object.

6. According to the digital twin-driven robotic arm reinforcement learning control platform and construction method according to claim 5, step 6 specifically includes the following contents: S61: Install and configure the ML-Agents plug-in in Unity to create a training environment for the control of the digital robotic arm model, including setting the initial position and target position of the robotic arm, as well as the reward and penalty rules; S62: defining the joint angle, end position and target position information of the robot arm as an observation space, so that the digital robot arm model can perceive the current state and target of the robot arm; S63: setting the action space of the Agent, including the control input of each joint, including the angular velocity or angle adjustment, so that the digital robot arm model outputs an action according to the strategy at each time step to adjust the posture of the robot arm toward the target; S64: Define a reward function, design rewards and penalties based on the distance between the end of the robot arm and the target position, whether the target is successfully reached, and action efficiency factors, and encourage the digital robot arm model to approach the target in an efficient and stable manner; S65: Run ML-Agents training, use reinforcement learning algorithms to train the control strategy of the robot arm, and continuously optimize the performance of the agent in different scenarios; S66: Observe and debug the learning effect of the agent during the training process, and gradually improve the control accuracy and stability of the robot arm by adjusting hyperparameters, reward functions, or adding new constraints; S67: After the training is completed, the trained model is deployed in Unity to verify its effect in the real simulation environment to ensure that it can complete the target action stably and accurately; S68: If necessary, export the optimized model in ONNX format and deploy it on the ROS side to apply it to the physical robot arm, so that it can use the optimized control strategy in actual operation to achieve more efficient and precise motion control.

Images