Digital twin-driven robotic arm reinforcement learning control platform and construction method

Abstract

The present invention discloses a reinforcement learning control platform and construction method for a robotic arm driven by digital twins. The platform has three modules: virtual control of real, real control of virtual, and training optimization. First, through the virtual control of real function, the digital entity model controls the physical robotic arm to achieve real-time control; the real control of virtual function uses the sensor data feedback of the physical robotic arm to update the joint angle, speed and other states of the digital entity model in real time, so that the virtual model is synchronized with the physical model. In addition, the platform performs reinforcement learning training in the digital entity model through the optimization training module. The trained model can be placed in the physical model for testing, and finally a complete optimization training strategy is formed to improve its operating efficiency and accuracy. The platform supports 3D simulation and dynamic data update to achieve high-fidelity digital twins of robotic arms. It can be widely used in industrial automation, remote control, intelligent manufacturing and other fields. It can improve control efficiency, optimize operating procedures, and enhance the safety and stability of robotic arm operations.

Description

Technical Field

[0001] The present invention relates to the fields of intelligent manufacturing and robotics technology, and in particular to a robotic arm reinforcement learning control platform and construction method based on digital twin drive. Background Art

[0002] In modern industry and intelligent manufacturing, robotic arms are widely used in a variety of complex automation tasks, such as welding, assembly, and handling. However, traditional robotic arm control methods typically rely on preprogrammed paths and fixed control strategies, which are unable to flexibly respond to real-time changes in environmental conditions and operational requirements. Furthermore, robotic arms may encounter various uncertainties in practical applications, such as sensor errors, mechanical deviations, and external disturbances, which can easily affect the control accuracy and operational stability of the robotic arm.

[0003] Digital twin technology, a cutting-edge technology that integrates the virtual and the real, has attracted widespread attention in the manufacturing industry in recent years. By constructing a digital robotic arm model that is consistent with the physical device, the digital twin can reflect the state of the physical system in real time, enabling real-time monitoring, simulation, and prediction of the physical device. In the field of robotic arm control, in particular, a control platform based on the digital twin can accurately simulate the robot's motion state using the digital robotic arm model, further improving control accuracy and response speed.

[0004] Existing digital twin systems typically implement one-way interaction from virtual to physical (virtual controlling the physical) or physical to virtual (physical controlling the virtual), but these systems have several operational deficiencies. For example, virtual control of the physical requires complex communication and control algorithms, while physical control of the virtual requires efficient data transmission and update mechanisms to ensure high consistency between the digital robotic arm model and the physical model. Furthermore, traditional methods lack effective training methods for operational optimization, making it difficult to dynamically adjust control strategies based on the actual environment, reducing the robotic arm's operational efficiency. Summary of the Invention

[0005] In response to the above-mentioned technical deficiencies, the purpose of the present invention is to provide a digital twin-driven robotic arm reinforcement learning control platform and construction method, which combines three functional modules: virtual control of real, real control of virtual, and training optimization, aiming to build an efficient, real-time, and accurate robotic arm control system. The virtual control of the real module realizes remote control of the physical robotic arm, and the real control of the virtual module realizes real-time feedback and synchronous update of the physical robotic arm state. At the same time, with the help of reinforcement learning to optimize the control strategy, the operating efficiency and accuracy of the robotic arm are improved. This innovative control platform can be widely used in industrial automation, remote control, and intelligent manufacturing, effectively solving the limitations of traditional control methods.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] The present invention provides a digital twin-driven robotic arm reinforcement learning control platform and construction method. The control platform includes a virtual-control-real module, a real-control-virtual module, and a training optimization module. The virtual-control-real module implements remote real-time control of a physical robotic arm through a digital robotic arm model. The real-control-virtual module is used to provide real-time feedback on the state of the physical robotic arm. By collecting real-time data from various sensors arranged on the robotic arm, including joint angles, speeds, and position information, these data are synchronously transmitted to the digital robotic arm model to ensure that the motion state of the digital robotic arm model is completely consistent with that of the physical robotic arm. The training optimization module automatically generates an optimized control strategy by performing reinforcement learning training in the digital robotic arm model and applies it to the physical robotic arm. The construction method of the control platform includes the following steps:

[0008] Step 1. Import the URDF model file and introduce the robotic arm model into Unity through the URDF Importer to form a digital robotic arm model;

[0009] Step 2: Build a digital environment in Unity that matches the actual environment;

[0010] Step 3. Configure the communication environment with ROS in Unity;

[0011] Step 4: Build a virtual-to-real module. Specifically, monitor the / joint_states topic of the physical robotic arm, synchronize the joint states to the Unity model, and implement physical control of the digital robotic arm.

[0012] Step 5: Build a real-control virtual module. Specifically, the digital robotic arm model sends the joint angles and desired end pose to ROS MoveIt, obtains the trajectory and sends it to the digital robotic arm model. At the same time, the physical robotic arm is controlled to execute, realizing virtual control of the physical object.

[0013] Step 6: Build a training optimization module. Specifically, use ML-Agents for reinforcement learning training to optimize the robotic arm control strategy.

[0014] Preferably, step 1 specifically includes the following contents:

[0015] S11. Install the URDF Importer plugin in Unity’s Package Manager.

[0016] S12. Import the URDF file and related resource files (such as mesh files in STL or DAE format) of the Zhongke Shengu ROCR6 robotic arm into the Assets folder of Unity.

[0017] S13. Select URDF Importer > Import URDF in the Unity menu to load the ROCR6 robotic arm model into the Unity scene.

[0018] Preferably, step 2 specifically includes the following contents:

[0019] S21. Take precise measurements based on the physical layout and dimensions of the actual environment and create a 3D model that is consistent with the real environment;

[0020] S22. Set the physical parameters of objects in the Unity scene, including friction coefficient, collision volume, mass, and gravity, to ensure that the behavior of virtual objects is consistent with real objects, thereby achieving high-fidelity physical simulation effects.

[0021] Preferably, step 3 specifically includes the following contents:

[0022] S31. Import the ros-tcp-connector and unityroboticsvisualizations components through the Window / Package Manager on the Unity platform.

[0023] S32. After executing S31, Unity will have an additional option Robotics, which configures the ROS IP and ROS Port;

[0024] S33. Compile ROS-TCP-Endpoint into the ROS workspace.

[0025] Preferably, step 4 specifically includes the following contents:

[0026] S41: As shown in step 3, configure the ROS communication plug-in in Unity, subscribe to the / joint_states topic, and obtain real-time joint state data from the physical robotic arm;

[0027] S42: Listen to the / joint_states topic to receive joint angle information, and parse the information into specific values ​​for each joint;

[0028] S43: Mapping the parsed joint angle values ​​to the digital robotic arm model in Unity, and converting the joint state data into angle information of each joint in Unity to maintain data consistency;

[0029] S44: Use smooth interpolation algorithms to optimize the motion of the Unity robotic arm, reduce jitter caused by data transmission delays or update frequency differences, and ensure the smoothness and real-time performance of the digital robotic arm model's motion.

[0030] Preferably, step 5 specifically includes the following contents:

[0031] S51: Create and start a ROS service node in Unity, so that it can send the joint angles and desired end pose of the digital robotic arm model to the ROS end. The desired end pose will use the position of the ball in Unity as the target.

[0032] S52: Send the current joint angle of the digital robot arm and the position of the ball to the ROS end through the ROS service request;

[0033] S53: Listen to and receive the service request sent by Unity, extract the joint angle and expected end pose from the request, and process them in MoveIt;

[0034] S54: Use MoveIt to perform path planning on the received joint angles and target poses, and calculate a trajectory that takes the robotic arm from its current position to the target pose; the trajectory includes the joint angle and pose changes at each time step;

[0035] S55: Send the calculated trajectory as the response data of the ROS service back to the Unity end, so that the digital robotic arm model in Unity can perform the corresponding movement according to the planned trajectory;

[0036] S56: Receive the trajectory data returned by ROS in Unity, parse the joint angles of each time step, and apply them to the joints of the Unity digital robotic arm model, so that the digital robotic arm model moves in the digital environment according to the trajectory planned by MoveIt;

[0037] S57: Apply the same trajectory to the physical robotic arm, making it perform the same movements as the Unity digital robotic arm model, achieving synchronized movement of virtual control of the physical object.

[0038] Preferably, step 6 specifically includes the following contents:

[0039] S61: Install and configure the ML-Agents plugin in Unity to create a training environment for controlling the digital robotic arm model, including setting the initial and target positions of the robotic arm, as well as reward and penalty rules.

[0040] S62: defining the joint angles, end position, and target position information of the robotic arm as an observation space, so that the digital robotic arm model can perceive the current state and target of the robotic arm;

[0041] S63: Setting the action space of the agent, including the control input of each joint, including the angular velocity or angle adjustment, so that the digital robotic arm model outputs an action according to the strategy at each time step and adjusts the posture of the robotic arm toward the target;

[0042] S64: Define the reward function, design rewards and penalties based on the distance between the end of the robotic arm and the target position, whether it successfully reaches the target, and action efficiency factors, and encourage the digital robotic arm model to approach the target in an efficient and stable manner.

[0043] S65: Run ML-Agents training, using reinforcement learning algorithms (such as PPO or SAC) to train the control strategy of the robotic arm and continuously optimize the agent's performance in different scenarios.

[0044] S66: Observe and debug the learning effect of the agent during the training process, and gradually improve the control accuracy and stability of the robotic arm by adjusting hyperparameters, reward functions, or adding new constraints.

[0045] S67: After training is completed, the trained model is deployed in Unity to verify its effect in a real simulation environment to ensure that it can complete the target action stably and accurately.

[0046] S68: If necessary, export the optimized model in ONNX format and deploy it on the ROS side to be applied to the physical robotic arm, so that it can use the optimized control strategy in actual operation to achieve more efficient and precise motion control.

[0047] The beneficial effects of the present invention are:

[0048] (1) Synchronous update of virtual and real: Through the real-control virtual module, the platform can collect real-time status information such as the joint angle and speed of the physical robot arm and synchronously update it to the digital twin model, that is, the digital robot arm model, to achieve dynamic consistency between the virtual model and the physical model. This real-time feedback mechanism enables users to accurately monitor the operating status of the robot arm, helps to quickly detect and troubleshoot potential faults, and improves the reliability of the system.

[0049] (2) Optimizing training efficiency: The training optimization module uses a reinforcement learning algorithm to simulate training on the digital twin model. The generated optimized control strategy can be directly applied to the physical robot arm. Through a large number of training iterations in the virtual environment, the system can automatically optimize the control strategy, improving the robot arm's operational efficiency and adaptability in complex tasks. In this way, the optimized control strategy can achieve efficient and accurate action execution in actual operation, shortening the time cost of strategy adjustment.

[0050] (3) High-fidelity 3D simulation and dynamic data update: The platform is built on Unity and features high-fidelity 3D simulation and real-time data update capabilities. This feature ensures that the user's experience of operating the robotic arm in a virtual environment is consistent with actual operation, providing operators with intuitive and realistic control feedback, further improving operational safety and system adaptability.

[0051] (4) Wide Applicability and Reliability: The platform is applicable to fields such as industrial automation, intelligent manufacturing, and remote control. It improves the safety, stability, and operational efficiency of robotic arm control through efficient digital twin technology. The system has strong robustness and adaptability in complex environments, which can reduce risks during robotic arm operations, improve production efficiency, and optimize operational processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 This is a flow chart of the method for building a digital twin robotic arm control platform based on Unity provided in this embodiment;

[0054] Figure 2 It is the robotic arm model used in the specific implementation case;

[0055] Figure 3 It is the interaction between gazebo (virtual reality) and Unity virtual reality;

[0056] Figure 4 It is the interactive tracking of real objects and Unity virtual reality; DETAILED DESCRIPTION

[0057] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0058] like Figures 1 to 2 As shown, this embodiment provides a robotic arm reinforcement learning control platform and construction method based on digital twin drive.

[0059] The platform comprises three functional modules: virtual control of the real, real control of the virtual, and training optimization. With a digital twin model at its core, the platform combines virtual-reality interactive control to achieve robotic arm control, real-time data feedback, and intelligent strategy optimization, improving the robotic arm's control accuracy, operational efficiency, and adaptability in practical applications.

[0060] First, the virtual control real module realizes remote real-time control of the physical robotic arm through the digital robotic arm model;

[0061] Secondly, the real-control virtual module is used to realize real-time feedback on the state of the physical robotic arm. By collecting real-time data from various sensors arranged on the robotic arm, including joint angle, speed, and position information, these data are synchronously transmitted to the digital robotic arm model to ensure that the digital robotic arm model is completely consistent with the motion state of the physical robotic arm; arranging corresponding sensors to obtain corresponding data belongs to the existing technology and will not be described in detail.

[0062] Finally, the training optimization module automatically generates an optimized control strategy by performing reinforcement learning training in the digital robotic arm model and applies it to the physical robotic arm.

[0063] During training, the platform simulates various working environments and conditions, conducting multiple rounds of training and optimization on the control strategy to improve its adaptability and efficiency in complex tasks. After training, the optimized control strategy is deployed in a physical robotic arm and validated in real-world applications to ensure more efficient and precise control.

[0064] Built on Unity, this platform features high-precision 3D simulation and dynamic data update capabilities, making it widely applicable to industrial automation, remote control, intelligent manufacturing, and other fields. This invention not only significantly improves the control efficiency and operational stability of the robotic arm, but also optimizes the control strategy through reinforcement learning, effectively enhancing the system's adaptability and robustness, enabling safe and efficient robotic arm operations in complex environments.

[0065] The method for constructing the control platform includes the following steps:

[0066] Step 1: Import the URDF model file and use the URDF Importer to introduce the robotic arm model into Unity to form a digital robotic arm model. Specifically:

[0067] S11. Install the URDF Importer plugin in Unity’s Package Manager.

[0068] S12. Import the URDF file and related resource files (such as mesh files in STL or DAE format) of the Zhongke Shengu ROCR6 robotic arm into the Assets folder of Unity.

[0069] S13. Select URDF Importer > Import URDF in the Unity menu to load the ROCR6 robotic arm model into the Unity scene.

[0070] Step 2: Build a digital environment in Unity that matches the actual environment;

[0071] S21. Take precise measurements based on the physical layout and dimensions of the actual environment and create a 3D model that is consistent with the real environment.

[0072] S22. Set the physical parameters of objects in the Unity scene, including friction coefficient, collision volume, mass, and gravity, to ensure that the behavior of virtual objects is consistent with real objects, thereby achieving high-fidelity physical simulation effects.

[0073] Step 3. Configure the communication environment with ROS in Unity;

[0074] S31. Import the ros-tcp-connector and unity robotics visualization components through the Window / Package Manager on the Unity platform.

[0075] S32. After executing S31, Unity will have an additional option Robotics, which configures the ROS IP and ROS Port;

[0076] S33. Compile ROS-TCP-Endpoint into the ROS workspace;

[0077] Before building a real physical platform, it is recommended to use Gazebo as a virtual physical robot arm for testing, and then transplant it to the physical platform.

[0078] Step 4: Build a virtual-to-real module. Specifically, monitor the / joint_states topic of the physical robotic arm, synchronize the joint states to the Unity model, and implement physical control of the digital robotic arm.

[0079] S41: As shown in step 3, configure the ROS communication plug-in in Unity, subscribe to the / joint_states topic, and obtain real-time joint state data from the physical robotic arm;

[0080] S42: Listen to the / joint_states topic to receive joint angle information, and parse the information into specific values ​​for each joint;

[0081] S43: Mapping the parsed joint angle values ​​to the digital robotic arm model in Unity, and converting the joint state data into angle information of each joint in Unity to maintain data consistency;

[0082] S44: Use smooth interpolation algorithms to optimize the motion of the Unity robotic arm, reduce jitter caused by data transmission delays or update frequency differences, and ensure the smoothness and real-time performance of the digital robotic arm model's motion.

[0083] Step 5: Build a real-control virtual module. Specifically, the digital robotic arm model sends the joint angles and desired end pose to ROS MoveIt, obtains the trajectory and sends it to the digital robotic arm model. At the same time, the physical robotic arm is controlled to execute, realizing virtual control of the physical object.

[0084] S51: Create and start a ROS service node in Unity, so that it can send the joint angles and desired end pose of the digital robotic arm model to the ROS end. The desired end pose will use the position of the ball in Unity as the target.

[0085] S52: The current joint angle of the digital robot arm and the position of the ball (as the desired end pose) are packaged and sent to the ROS end through a ROS service request;

[0086] S53: Listen to and receive the service request sent by Unity, extract the joint angle and expected end pose from the request, and process them in MoveIt;

[0087] S54: Use MoveIt to perform path planning on the received joint angles and target poses, and calculate a trajectory that takes the robotic arm from its current position to the target pose; the trajectory includes the joint angle and pose changes at each time step;

[0088] S55: Send the calculated trajectory as the response data of the ROS service back to the Unity end, so that the digital robotic arm model in Unity can perform the corresponding movement according to the planned trajectory;

[0089] S56: Receive the trajectory data returned by ROS in Unity, parse the joint angles of each time step, and apply them to the joints of the Unity digital robotic arm model, so that the digital robotic arm model moves in the digital environment according to the trajectory planned by MoveIt;

[0090] S57: Apply the same trajectory to the physical robotic arm, making it perform the same movements as the Unity digital robotic arm model, achieving synchronized movement of virtual control of the physical object.

[0091] Through steps 4 and 5, the virtual-to-real and real-to-virtual functions can be realized. The interactive tracking between the virtual and real parts using gazebo and Unity is as follows: Figure 3As shown in the figure, the joint movement range is -180° to 180°, and the maximum error is within 2°, which means the tracking effect is good. After completing steps 4 and 5, you can deploy it on a physical robotic arm for testing. The interactive tracking between the real object and Unity virtual reality is as follows: Figure 4 As shown, the maximum error is still within 2°.

[0092] Step 6: Build a training optimization module. Specifically, use ML-Agents for reinforcement learning training to optimize the robotic arm control strategy.

[0093] S61: Install and configure the ML-Agents plugin in Unity to create a training environment for controlling the digital robotic arm model, including setting the initial and target positions of the robotic arm, as well as reward and penalty rules.

[0094] S62: defining the joint angles, end position, and target position information of the robotic arm as an observation space, so that the digital robotic arm model can perceive the current state and target of the robotic arm;

[0095] S63: Setting the action space of the agent, including the control input of each joint, including the angular velocity or angle adjustment, so that the digital robotic arm model outputs an action according to the strategy at each time step and adjusts the posture of the robotic arm toward the target;

[0096] S64: Define the reward function, design rewards and penalties based on the distance between the end of the robotic arm and the target position, whether it successfully reaches the target, and action efficiency factors, and encourage the digital robotic arm model to approach the target in an efficient and stable manner.

[0097] S65: Run ML-Agents training, using reinforcement learning algorithms (such as PPO or SAC) to train the control strategy of the robotic arm and continuously optimize the agent's performance in different scenarios.

[0098] S66: Observe and debug the learning effect of the agent during the training process, and gradually improve the control accuracy and stability of the robotic arm by adjusting hyperparameters, reward functions, or adding new constraints.

[0099] S67: After training is completed, the trained model is deployed in Unity to verify its effect in a real simulation environment to ensure that it can complete the target action stably and accurately.

[0100] S68: If necessary, export the optimized model in ONNX format and deploy it on the ROS side to be applied to the physical robotic arm, so that it can use the optimized control strategy in actual operation to achieve more efficient and precise motion control.

[0101] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A digital twin-driven robotic arm reinforcement learning control platform and construction method, characterized by: The control platform includes a virtual-control-real module, a real-control-virtual module, and a training optimization module; the virtual-control-real module implements remote, real-time control of the physical robotic arm through a digital robotic arm model; the real-control-virtual module is used to provide real-time feedback on the physical robotic arm's status by collecting real-time data from various sensors arranged on the robotic arm, including joint angles, speeds, and position information, and synchronously transmitting this data to the digital robotic arm model to ensure that the motion state of the digital robotic arm model is completely consistent with that of the physical robotic arm; The training optimization module automatically generates an optimized control strategy by performing reinforcement learning training in a digital robotic arm model and applies it to the physical robotic arm. The control platform construction method includes the following steps: Step 1. Import the URDF model file and introduce the robotic arm model into Unity through the URDF Importer to form a digital robotic arm model; Step 2: Build a digital environment in Unity that matches the actual environment; Step 3. Configure the communication environment with ROS in Unity; Step 4: Build a virtual-to-real module. Specifically, monitor the / joint_states topic of the physical robotic arm, synchronize the joint states to the Unity model, and implement physical control of the digital robotic arm. Step 5: Build a real-control virtual module. Specifically, the digital robotic arm model sends the joint angles and desired end pose to ROS MoveIt, obtains the trajectory and sends it to the digital robotic arm model. At the same time, the physical robotic arm is controlled to execute, realizing virtual control of the physical object. Step 6: Build a training optimization module. Specifically, use ML-Agents for reinforcement learning training to optimize the robotic arm control strategy.

2. The digital twin-driven robotic arm reinforcement learning control platform and construction method according to claim 1 is characterized in that: Step 2 specifically includes the following: S21. Take precise measurements based on the physical layout and dimensions of the actual environment and create a 3D model that is consistent with the real environment; S22. Set the physical parameters of objects in the Unity scene, including friction coefficient, collision volume, mass, and gravity, to ensure that the behavior of virtual objects is consistent with real objects, thereby achieving high-fidelity physical simulation effects.

3. The digital twin-driven robotic arm reinforcement learning control platform and construction method according to claim 2 is characterized in that: Step 3 specifically includes the following: S31. Import the ros-tcp-connector and unityroboticsvisualizations components through the Window / Package Manager on the Unity platform. S32. After executing S31, Unity will have an additional option Robotics, which configures the ROS IP and ROS Port; S33. Compile ROS-TCP-Endpoint into the ROS workspace.

4. The digital twin-driven robotic arm reinforcement learning control platform and construction method according to claim 3 is characterized in that: Step 4 specifically includes the following: S41: As shown in step 3, configure the ROS communication plug-in in Unity, subscribe to the / joint_states topic, and obtain real-time joint state data from the physical robotic arm; S42: Listen to the / joint_states topic to receive joint angle information, and parse the information into specific values ​​for each joint; S43: Mapping the parsed joint angle values ​​to the digital robotic arm model in Unity, and converting the joint state data into angle information of each joint in Unity to maintain data consistency; S44: Use smooth interpolation algorithms to optimize the motion of the Unity robotic arm, reduce jitter caused by data transmission delays or update frequency differences, and ensure the smoothness and real-time performance of the digital robotic arm model's motion.

5. The digital twin-driven robotic arm reinforcement learning control platform and construction method according to claim 4 is characterized in that: Step 5 specifically includes the following: S51: Create and start a ROS service node in Unity, so that it can send the joint angles and desired end pose of the digital robotic arm model to the ROS end. The desired end pose will use the position of the ball in Unity as the target. S52: Send the current joint angle of the digital robot arm and the position of the ball to the ROS end through the ROS service request; S53: Listen to and receive the service request sent by Unity, extract the joint angle and expected end pose from the request, and process them in MoveIt; S54: Use MoveIt to perform path planning on the received joint angles and target poses, and calculate a trajectory that takes the robotic arm from its current position to the target pose; the trajectory includes the joint angle and pose changes at each time step; S55: Send the calculated trajectory as the response data of the ROS service back to the Unity end, so that the digital robotic arm model in Unity can perform the corresponding movement according to the planned trajectory; S56: Receive the trajectory data returned by ROS in Unity, parse the joint angles of each time step, and apply them to the joints of the Unity digital robotic arm model, so that the digital robotic arm model moves in the digital environment according to the trajectory planned by MoveIt; S57: Apply the same trajectory to the physical robotic arm, making it perform the same movements as the Unity digital robotic arm model, achieving synchronized movement of virtual control of the physical object.

6. According to the digital twin-driven robotic arm reinforcement learning control platform and construction method according to claim 5, step 6 specifically includes the following content: S61: Install and configure the ML-Agents plugin in Unity to create a training environment for controlling the digital robotic arm model, including setting the initial and target positions of the robotic arm, as well as reward and penalty rules. S62: defining the joint angles, end position, and target position information of the robotic arm as an observation space, so that the digital robotic arm model can perceive the current state and target of the robotic arm; S63: Setting the action space of the agent, including the control input of each joint, including the angular velocity or angle adjustment, so that the digital robotic arm model outputs an action according to the strategy at each time step and adjusts the posture of the robotic arm toward the target; S64: Define a reward function, design rewards and penalties based on the distance between the end of the manipulator and the target position, whether it successfully reaches the target, and action efficiency factors, and encourage the digital manipulator model to approach the target in an efficient and stable manner; S65: Run ML-Agents training, using reinforcement learning algorithms to train the control strategy of the robotic arm and continuously optimize the agent's performance in different scenarios; S66: Observe and debug the learning effect of the agent during training, and gradually improve the control accuracy and stability of the robot arm by adjusting hyperparameters, reward functions, or adding new constraints; S67: After the training is completed, the trained model is deployed in Unity to verify its effect in the real simulation environment to ensure that it can complete the target action stably and accurately; S68: If necessary, export the optimized model in ONNX format and deploy it on the ROS side to be applied to the physical robotic arm, so that it can use the optimized control strategy in actual operation to achieve more efficient and precise motion control.

Images