Robot autonomous programming system and method based on reinforcement learning

Abstract

The embodiment of the invention provides a robot autonomous programming system and method based on reinforcement learning, and belongs to the technical field of robot automation. The system comprises an autonomous programming engine module, a simulation environment training module, a real environment migration module, a sensor feedback and data processing module, and a controller and execution module. The autonomous programming engine module is used for converting a high-level task instruction into a low-level action program; the simulation environment training module is used for providing a virtual training platform and reinforcement learning training; the real environment migration module is used for migrating the strategy learned from the simulation environment to the real environment; the sensor feedback and data processing module is used for collecting and processing various sensor data of the robot; and the controller and execution module is used for receiving the instruction from the autonomous programming engine module and controlling the robot to execute the task. According to the method, the complexity of robot programming can be reduced, and the flexibility and applicability of a robot system are improved.

Description

Technical Field

[0001] The present application relates to the field of robot automation technology, and in particular to a robot autonomous programming system and method based on reinforcement learning. Background Art

[0002] In recent years, with the rapid development of artificial intelligence and robotics, robots have been increasingly used in industrial manufacturing, service industry, medical treatment, logistics and other fields. Traditional robotic systems mainly rely on pre-programmed fixed instruction sets, which are written by professional engineers using specific programming languages ​​(such as KRL, RAPID, URScript, etc.). These programming languages ​​usually have complex syntax and steep learning curves, requiring deep professional knowledge and are difficult for non-professionals to master.

[0003] Current methods of autonomous robot learning and programming have several limitations:

[0004] Sim-to-Real Gap Problem: Existing robot autonomous learning systems based on reinforcement learning often rely on a large amount of simulated training in virtual environments. However, virtual environments cannot fully and realistically reproduce the complex physical properties, dynamic changes, and uncertainties in the real world. Therefore, the strategies trained by robots in simulated environments are difficult to directly transfer to real environments, resulting in unstable performance in practical applications.

[0005] Low learning efficiency: Reinforcement learning algorithms usually require a lot of training data and time, especially in tasks with high dimensions and continuous action spaces, where convergence is slow and practical applications are limited.

[0006] Safety issues: During the exploration process of reinforcement learning, the robot may perform unknown or unsafe actions, which may cause damage to equipment or injury to personnel.

[0007] Unfriendly human-computer interaction: Although teaching programming has lowered the professional threshold for programming, it still takes a lot of time to teach and debug in complex tasks and environments.

[0008] Limited autonomous programming capabilities: Although teaching programming has lowered the threshold for robot programming and enabled non-professionals to guide robots to perform tasks, existing systems are still highly dependent on manual teaching. This reliance on manual intervention has greatly increased the complexity of system use and reduced the degree of robot automation. Summary of the invention

[0009] The main purpose of the embodiments of the present application is to provide a robot autonomous programming system and method based on reinforcement learning.

[0010] The technical solution adopted by the present invention is:

[0011] On the one hand, an embodiment of the present invention provides a robot autonomous programming system based on reinforcement learning, the system comprising an autonomous programming engine module, a simulation environment training module, a real environment migration module, a sensor feedback and data processing module, a controller and an execution module;

[0012] The autonomous programming engine module is used to convert high-level task instructions into low-level action programs;

[0013] The simulation environment training module is used to provide a virtual training platform and reinforce learning training;

[0014] The real environment migration module is used to migrate the strategy learned from the simulation environment to the real environment;

[0015] The sensor feedback and data processing module is used to collect and process various sensor data of the robot;

[0016] The controller and execution module are used to receive instructions from the autonomous programming engine module and control the robot to perform tasks.

[0017] Further, the autonomous programming engine module includes a behavior library, a high-level task instruction set submodule, and a reinforcement learning strategy generation submodule;

[0018] The behavior library is used to store basic actions performed by the robot, and the basic actions include moving, grasping, and rotating;

[0019] The high-level task instruction set submodule is used to simplify the instruction set input by the user;

[0020] The reinforcement learning strategy generation submodule is used to automatically learn the task strategy through the reinforcement learning algorithm to generate an optimized action sequence.

[0021] Furthermore, the simulation environment training module includes a simulation platform, a reinforcement learning training submodule, and a virtual environment optimization submodule;

[0022] The simulation platform is used to simulate the interaction between the robot and the physical environment and create a task scenario;

[0023] The reinforcement learning training submodule is used to optimize the robot task execution strategy using the reinforcement learning algorithm and adjust the robot behavior through the reward feedback mechanism;

[0024] The virtual environment optimization submodule is used to enhance the diversity of the simulation environment through domain randomization technology.

[0025] Furthermore, the real environment migration module includes a sensor data acquisition submodule, a migration learning submodule, and a strategy fine-tuning submodule;

[0026] The sensor data acquisition submodule is used to obtain online environmental information through sensors; the sensors include visual sensors, force sensors, and laser radars;

[0027] The transfer learning submodule is used to adjust the strategy obtained in the simulation environment through the transfer learning technology;

[0028] The strategy fine-tuning submodule is used to perform adjustments through online training in a real environment.

[0029] Furthermore, the sensor feedback and data processing module includes a multi-sensor fusion submodule, a data preprocessing submodule, and a state estimation submodule;

[0030] The multi-sensor fusion submodule is used to integrate data from sensors to form a comprehensive perception of the environment;

[0031] The data preprocessing submodule is used to perform denoising, filtering and standardization on the sensor data;

[0032] The state estimation submodule is used to estimate the current position and speed state of the robot through the Kalman filter algorithm.

[0033] Further, the controller and execution module includes a motion controller submodule, a path planning and execution submodule, and a safety control mechanism submodule;

[0034] The motion controller submodule is used to control the motion of the robot;

[0035] The path planning and execution submodule is used to perform path planning according to the learned strategy;

[0036] The safety control mechanism submodule is used to control the robot to avoid collision during mission execution.

[0037] Furthermore, the robot autonomous programming system based on reinforcement learning also includes a user teaching optimization module and a human-computer interaction layer module; the user teaching optimization module is used to collect the teaching data of the operator and optimize the action strategy of the robot to perform tasks in combination with supervised learning;

[0038] The user teaching optimization module includes a teaching data recording submodule, a supervised learning submodule, and a user-friendly interface submodule; the human-computer interaction layer module is used to provide an operation interface;

[0039] The teaching data recording submodule is used to record the action data when the operator guides the robot through manual operation or teaching equipment;

[0040] The supervised learning submodule is used to use the teaching data for supervised learning to generate a strategy for the robot;

[0041] The user-friendly interface submodule is used to provide a user interface;

[0042] The human-computer interaction layer module includes a visual monitoring interface submodule, a task setting and modification submodule, and an alarm and prompt module submodule;

[0043] The visual monitoring interface submodule is used to display robot status, task progress, and sensor data information;

[0044] The task setting and modification submodule is used to define task objectives and execution parameters;

[0045] The alarm and prompt module submodule is used to issue an alarm when an abnormality occurs in a task.

[0046] Furthermore, the robot autonomous programming system based on reinforcement learning also includes a communication and synchronization module; the communication and synchronization module is used to coordinate the collaboration of multiple robots;

[0047] The communication and synchronization module includes a high-speed communication network submodule and a time synchronization mechanism submodule;

[0048] The high-speed communication network submodule is used to realize online data transmission between various devices through industrial Ethernet or wireless network;

[0049] The time synchronization mechanism submodule is used to perform motion synchronization when multiple robots collaborate through a precise time protocol.

[0050] On the other hand, an embodiment of the present invention further provides a robot autonomous programming method based on reinforcement learning, the method comprising the following steps:

[0051] Through the autonomous programming engine module, high-level task instructions are converted into low-level action programs;

[0052] Through the simulation environment training module, a virtual training platform and reinforcement learning training are provided;

[0053] Through the real environment transfer module, the strategies learned from the simulation environment are transferred to the real environment;

[0054] Collect and process various sensor data of the robot through the sensor feedback and data processing module;

[0055] Through the controller and the execution module, the instructions from the autonomous programming engine module are received, and the robot is controlled to perform tasks.

[0056] Furthermore, the robot autonomous programming method based on reinforcement learning also includes the following steps:

[0057] Get task information;

[0058] Constructing a virtual task environment for training according to the task information, and obtaining training result information;

[0059] Perform transfer learning based on the training result information to obtain task strategy transfer information;

[0060] Acquire mission strategy optimization information according to the mission strategy migration information and sensor collection data;

[0061] According to the task strategy optimization information, the robot autonomous programming is completed.

[0062] The embodiments of the present application include at least the following beneficial effects: The present application provides a robot autonomous programming system and method based on reinforcement learning. The system of the present invention includes an autonomous programming engine module, a simulation environment training module, a real environment migration module, a sensor feedback and data processing module, a controller and an execution module; the autonomous programming engine module is used to convert high-level task instructions into low-level action programs; the simulation environment training module is used to provide a virtual training platform and reinforcement learning training; the real environment migration module is used to migrate the strategies learned from the simulation environment to the real environment; the sensor feedback and data processing module is used to collect and process various types of sensor data of the robot; the controller and the execution module are used to receive instructions from the autonomous programming engine module and control the robot to perform tasks. The present invention can reduce the complexity of robot programming and improve the flexibility and applicability of the robot system. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 This is a diagram of the overall operation module architecture of a robot autonomous programming system based on reinforcement learning provided by an embodiment of the present invention;

[0064] Figure 2 It is an overall operation flow chart provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0066] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0067] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0069] The present invention takes into account the problems of high complexity, low learning efficiency, poor adaptability, and reliance on manual programming and teaching in existing robot programming. By introducing technologies such as reinforcement learning, autonomous programming engine, simulated environment training, transfer learning, and user teaching optimization, the present invention provides a simplified and automated robot programming method, which enables the robot to autonomously learn and generate action programs to adapt to changes in different tasks and environments. The system acquires environmental information in real time through multi-sensor fusion, combined with efficient control and execution modules, to achieve efficient completion of tasks and safe collaboration. The system is designed with an intuitive and friendly user interface to reduce the difficulty of operation, and is suitable for industrial manufacturing, service robots and other fields. The present invention not only significantly reduces the complexity of robot programming, but also improves the flexibility and applicability of the robot system, solving many bottleneck problems in the prior art.

[0070] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0071] 1) DDPG (Deep Deterministic Policy Gradient) is a deep reinforcement learning algorithm in reinforcement learning (RL).

[0072] 2) PPO (Proximal Policy Optimization) is a deep reinforcement learning algorithm in reinforcement learning.

[0073] 3) Gazebo, a software platform for robotics simulation and development.

[0074] 4) Webots, a software platform for robot simulation and development.

[0075] 5)PTP (Precision Time Protocol) is a network protocol.

[0076] The embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0077] On the one hand, an embodiment of the present invention provides a robot autonomous programming system based on reinforcement learning, the system includes an autonomous programming engine module 1, a simulation environment training module 2, a real environment migration module 3, a sensor feedback and data processing module 4, and a controller and execution module 5;

[0078] The autonomous programming engine module 1 is used to convert high-level task instructions into low-level action programs;

[0079] The simulation environment training module 2 is used to provide a virtual training platform and reinforce learning training;

[0080] The real environment transfer module 3 is used to transfer the strategies learned from the simulation environment to the real environment;

[0081] The sensor feedback and data processing module 4 is used to collect and process various sensor data of the robot;

[0082] The controller and execution module 5 is used to receive instructions from the autonomous programming engine module 1 and control the robot to perform tasks.

[0083] The autonomous programming engine module 1 disclosed in the embodiment of the present invention includes a behavior library, a high-level task instruction set submodule, and a reinforcement learning strategy generation submodule;

[0084] The behavior library is used to store the basic actions performed by the robot, including moving, grabbing, and rotating;

[0085] The high-level task instruction set submodule is used to simplify the instruction set input by the user;

[0086] The reinforcement learning strategy generation submodule is used to automatically learn task strategies through reinforcement learning algorithms and generate optimized action sequences.

[0087] The simulation environment training module 2 disclosed in the embodiment of the present invention includes a simulation platform, a reinforcement learning training submodule, and a virtual environment optimization submodule;

[0088] The simulation platform is used to simulate the interaction between the robot and the physical environment and create mission scenarios;

[0089] The reinforcement learning training submodule is used to optimize the robot's task execution strategy using the reinforcement learning algorithm and adjust the robot's behavior through the reward feedback mechanism;

[0090] The virtual environment optimization submodule is used to enhance the diversity of the simulation environment through domain randomization technology.

[0091] The real environment migration module 3 disclosed in the embodiment of the present invention includes a sensor data acquisition submodule, a migration learning submodule, and a strategy fine-tuning submodule;

[0092] The sensor data acquisition submodule is used to obtain online environmental information through sensors; the sensors include visual sensors, force sensors, and laser radars;

[0093] The transfer learning submodule is used to adjust the strategy acquired in the simulation environment through transfer learning technology;

[0094] The policy fine-tuning submodule is used to adjust the policy through online training in the real environment.

[0095] The sensor feedback and data processing module 4 disclosed in the embodiment of the present invention includes a multi-sensor fusion submodule, a data preprocessing submodule, and a state estimation submodule;

[0096] The multi-sensor fusion submodule is used to integrate data from sensors to form a comprehensive perception of the environment;

[0097] The data preprocessing submodule is used to denoise, filter and standardize the sensor data;

[0098] The state estimation submodule is used to estimate the current position and velocity state of the robot through the Kalman filter algorithm.

[0099] The controller and execution module 5 disclosed in the embodiment of the present invention includes a motion controller submodule, a path planning and execution submodule, and a safety control mechanism submodule;

[0100] The motion controller submodule is used to control the robot's motion;

[0101] The path planning and execution submodule is used to plan the path according to the learned strategy;

[0102] The safety control mechanism submodule is used to control the robot to avoid collision during mission execution.

[0103] The robot autonomous programming system based on reinforcement learning disclosed in the embodiment of the present invention further includes a user teaching optimization module 6 and a human-machine interaction layer module 7; the user teaching optimization module 6 is used to collect the teaching data of the operator and optimize the action strategy of the robot to perform the task in combination with supervised learning;

[0104] The user teaching optimization module 6 includes a teaching data recording submodule, a supervised learning submodule, and a user-friendly interface submodule; the human-computer interaction layer module 7 is used to provide an operation interface;

[0105] The teaching data recording submodule is used to record the action data when the operator guides the robot through manual operation or teaching equipment;

[0106] The supervised learning submodule is used to use the teaching data for supervised learning and generate strategies for the robot;

[0107] The user-friendly interface submodule is used to provide a user interface;

[0108] The human-computer interaction layer module 7 includes a visual monitoring interface submodule, a task setting and modification submodule, and an alarm and prompt module submodule;

[0109] The visual monitoring interface submodule is used to display robot status, task progress, and sensor data information;

[0110] The task setting and modification submodule is used to define task objectives and execution parameters;

[0111] The alarm and prompt module submodule is used to issue an alarm when an abnormality occurs in a task.

[0112] The robot autonomous programming system based on reinforcement learning disclosed in the embodiment of the present invention further includes a communication and synchronization module 8; the communication and synchronization module 8 is used to coordinate the cooperation of multiple robots;

[0113] The communication and synchronization module 8 includes a high-speed communication network submodule and a time synchronization mechanism submodule;

[0114] The high-speed communication network submodule is used to realize online data transmission between devices through industrial Ethernet or wireless network;

[0115] The time synchronization mechanism submodule is used to synchronize the actions of multiple robots during collaboration through a precise time protocol.

[0116] On the other hand, an embodiment of the present invention further discloses a robot autonomous programming method based on reinforcement learning, the method comprising the following steps:

[0117] S100, converting high-level task instructions into low-level action programs through the autonomous programming engine module 1;

[0118] S200, providing a virtual training platform and reinforcement learning training through a simulated environment training module 2;

[0119] S300, migrating the strategy learned from the simulation environment to the real environment through the real environment migration module 3;

[0120] S400, collecting and processing various sensor data of the robot through the sensor feedback and data processing module 4;

[0121] S500, receiving instructions from the autonomous programming engine module 1 through the controller and execution module 5, and controlling the robot to perform tasks.

[0122] The robot autonomous programming method based on reinforcement learning disclosed in the embodiment of the present invention also includes the following steps:

[0123] S600, obtaining task information;

[0124] S700, constructing a virtual task environment for training according to the task information, and obtaining training result information;

[0125] S800, performing transfer learning according to the training result information to obtain task strategy transfer information;

[0126] S900, obtaining task strategy optimization information according to task strategy migration information and sensor collection data;

[0127] S1000, optimizes information based on task strategies and completes autonomous robot programming.

[0128] As an optional implementation, the robot autonomous programming system based on reinforcement learning in the embodiment of the present invention aims to enable the robot to autonomously generate action programs through reinforcement learning technology and simplified programming, reduce dependence on manual programming, and improve task adaptability and system flexibility. Figure 1 ,The system architecture includes the following main modules:

[0129] 1. Autonomous programming engine

[0130] The autonomous programming engine is the core part of the present invention, which is used to replace the traditional robot programming method, provide a simplified instruction set and the ability to generate action programs based on reinforcement learning. This module generates an optimized action sequence to complete the task by parsing high-level task instructions and combining reinforcement learning algorithms.

[0131] Behavior library: stores commonly used basic robot actions (such as moving, grasping, rotating, etc.), providing basic operation units for the learning process.

[0132] High-level task instruction set: The operator only needs to define the task objectives, and the system describes the task through a simplified instruction set, avoiding complex low-level programming.

[0133] Reinforcement learning strategy generation: The system automatically learns and generates task execution strategies through reinforcement learning algorithms (such as DDPG and PPO), and the robot gradually derives specific action programs from high-level instructions.

[0134] 2. Simulation environment training module

[0135] This module provides a virtual training environment for the robot for the initial training of reinforcement learning. The robot learns the task execution strategy by interacting with the virtual physical world in the simulation environment, reducing the trial and error cost in the real environment and speeding up the training efficiency.

[0136] Simulation platform: Use simulation platforms such as Gazebo and Webots to simulate the physical characteristics and task scenarios of the robot.

[0137] Reinforcement learning training module: The robot is trained using a reinforcement learning algorithm in a simulation environment, and continuously optimizes its action strategy through a state-action-reward feedback mechanism.

[0138] Virtual environment optimization: Through technologies such as domain randomization, the simulation environment covers the uncertainties in various real-life scenarios, narrows the gap between simulation and reality, and enhances the migration capability of strategies.

[0139] 3. Real environment migration module

[0140] This module is used to transfer the training results in the simulation environment to the real robot to ensure the robot's task execution effect in the real environment. This module fine-tunes and optimizes the strategy through sensor feedback and transfer learning technology.

[0141] Sensor data collection: The robot uses vision, force perception, lidar and other sensors to perceive the state of the real environment in real time.

[0142] Transfer learning: Using sensor feedback, calibrate and optimize policies learned from simulation to adapt them to the physics of the real scene.

[0143] Strategy fine-tuning: Through a small amount of online training or adjustment, the model's performance in the real environment can be further optimized to improve task completion.

[0144] 4. User teaching optimization module

[0145] The user teaching optimization module allows operators to quickly guide the robot to learn new tasks through manual guidance, enhancing the usability of the system. Especially when the robot faces a completely new task, it can quickly learn key actions through teaching.

[0146] Teaching data recording: The operator manually controls the robot or uses the teaching device, and the system records the key data of the operation process as the basis for learning.

[0147] Supervised learning: In the initial task execution, teaching data is used for supervised learning to help the robot quickly generate initial strategies.

[0148] User-friendly interface: The intuitive user interface allows the operator to view and adjust the robot tasks during the teaching process and guide the robot to optimize the movements at any time.

[0149] 5. Sensor feedback and data processing module

[0150] This module is responsible for collecting the robot's sensor data and processing the data as an important basis for task execution and strategy optimization. Sensor feedback enables the robot to perceive environmental changes and make corresponding adjustments.

[0151] Multi-sensor fusion: Integrate feedback from vision, force perception, lidar and other sensors to obtain comprehensive environmental information.

[0152] Data preprocessing: De-noise, filter and standardize sensor data to ensure data accuracy and consistency.

[0153] State estimation: The current state of the robot (such as position, speed, and posture) is estimated through algorithms such as Kalman filtering to facilitate dynamic adjustment of execution strategies.

[0154] 6. Controller and execution module

[0155] The controller and execution module is responsible for receiving policy instructions from the autonomous programming engine and controlling the robot's hardware to perform specific task operations. This module needs to ensure the accuracy and safety of the robot's movements.

[0156] Motion controller: controls the movement of robot actuators such as manipulator arms and joints to ensure that each step in the action sequence is completed.

[0157] Path planning and execution: Based on the learned strategy, the controller plans and executes the robot's path to ensure efficient and accurate completion of the task.

[0158] Safety control mechanism: When the robot is performing tasks, a safety mechanism is established to prevent the robot from performing dangerous actions and ensure the safety of equipment and personnel.

[0159] 7. Communication and synchronization module

[0160] This module is used to coordinate the communication between various subsystems to ensure real-time data transmission and task synchronization, which is especially important in the case of multi-robot collaboration or robot interaction with external systems.

[0161] High-speed communication network: Real-time data transmission between robots and controllers, sensors and other devices is achieved through industrial Ethernet or wireless communication.

[0162] Time synchronization mechanism: Through the Precision Time Protocol (PTP) or other synchronization methods, time synchronization in multi-robot systems or multi-task execution is ensured to ensure seamless collaborative work.

[0163] 8. Human-computer interaction layer

[0164] The human-computer interaction layer is the external interface of the system, which is used to provide monitoring, task setting, teaching and system management functions to operators. This layer is designed with an intuitive and friendly user interface, which simplifies the operation process and lowers the threshold for using the system.

[0165] Visual monitoring interface: Real-time display of key information such as robot status, task progress, sensor data, etc., making it easier for operators to understand the system operation status.

[0166] Mission setting and modification: Allows operators to define mission objectives and execution parameters through a simplified high-level instruction set without writing complex programs.

[0167] Alarm and prompt module: When the system detects an abnormality or a problem occurs in task execution, an alarm prompt is automatically issued, and the operator can manually intervene or make adjustments through the interface.

[0168] As an optional implementation, the robot autonomous programming system based on reinforcement learning of the present invention adopts technologies such as reinforcement learning, autonomous programming engine, virtual environment training, transfer learning, and sensor feedback optimization to solve the problems of complex programming, low efficiency, and poor adaptability of existing robot systems. The technical implementation process of each module is described in detail below:

[0169] 1. Autonomous programming engine implementation

[0170] The autonomous programming engine is the core module of this system. It is responsible for converting high-level task descriptions into executable action programs, avoiding the traditional low-level programming requirements.

[0171] Behavior library construction:

[0172] The autonomous programming engine first builds a behavior library that contains the robot's basic action units, such as "move to a specified location", "grasp an object", "rotate", etc. These basic actions provide a basic action space for reinforcement learning, ensuring that the robot can combine these basic actions to perform complex tasks during training.

[0173] High level instruction analysis:

[0174] The user inputs a task instruction through the human-computer interaction interface, such as "move an object from point A to point B". After receiving the high-level instruction, the autonomous programming engine uses the instruction parsing algorithm to convert it into an action sequence in the behavior library. At the same time, the parsed task goal will serve as a source of reward signals for reinforcement learning to guide the robot to generate strategies.

[0175] Reinforcement Learning Policy Generation:

[0176] The system uses a deep reinforcement learning algorithm (such as DDPG or PPO) to optimize the robot's strategy for completing tasks through a state-action-reward cycle. The input of reinforcement learning is the robot state and the environment state, and the output is a specific action combination in the behavior library. The system continuously updates the strategy based on the task completion status and ultimately outputs the optimal action sequence to complete the task.

[0177] 2. Implementation of Simulation Environment Training

[0178] The simulated environment training module is used to initially train the robot to reduce the trial and error cost in the real environment. The robot is trained extensively in the virtual environment through the reinforcement learning algorithm to gradually optimize the task execution strategy.

[0179] Virtual environment construction:

[0180] Use simulation platforms (such as Gazebo and Webots) to create a high-fidelity virtual environment to simulate the physical model and working scene of the robot in reality. The simulation environment includes factors such as the physical properties of objects, gravity, friction, etc. to ensure that it is as consistent as possible with the real world.

[0181] Reinforcement Learning Training:

[0182] In a virtual environment, the robot performs training tasks through reinforcement learning. At the beginning of training, the system initializes the robot state and randomly generates actions. The robot interacts with the environment and continuously updates its strategy to gradually learn the best execution path. In each round of training, the system measures the effectiveness of the robot's task completion, such as task completion time and accuracy, through a set reward function. Through experience replay and policy gradient updates, the system optimizes the effect and efficiency of learning.

[0183] Parallel training and acceleration:

[0184] To improve training efficiency, the system uses multi-threaded parallel training, and multiple robots are trained simultaneously in different virtual scenes to increase the amount of data samples. The distributed algorithm in reinforcement learning is used to accelerate strategy optimization and reduce training time.

[0185] 3. Real environment migration implementation

[0186] After completing the policy training in the simulation environment, the system will transfer the learned policy to the real environment for application. Due to the differences between the simulation environment and the real environment, the system will fine-tune and optimize through sensor feedback and transfer learning technology.

[0187] Transfer learning strategy:

[0188] To reduce the impact of the "Sim-to-Real Gap", the system uses transfer learning technology. Through the domain adaptation method, the robot fine-tunes the strategy learned in the virtual environment in the real environment to ensure that the strategy can be effectively executed in the real scene. The system adjusts the physical properties of the environment to make it easier to apply the results of simulation training in the real environment.

[0189] Sensor Feedback Calibration:

[0190] In a real environment, the robot obtains the state of the environment in real time through multiple sensors such as vision, force perception, and lidar. The system evaluates the robot's task execution status based on sensor feedback, and introduces this feedback information into the strategy for online fine-tuning. For example, if the force feedback perceived by the robot during a grasping operation does not match expectations, the system will automatically adjust the grasping force or action path.

[0191] Online training and optimization:

[0192] In real scenarios, the system combines sensor data to conduct a small amount of online training, fine-tune strategy parameters, and improve the accuracy of task execution. This process ensures that the robot can still maintain efficient and accurate task execution in different actual operating environments.

[0193] 4. User teaching optimization implementation

[0194] The user teaching module is an optimization auxiliary function of the system, which allows the operator to quickly help the robot adapt to new tasks through manual guidance. The operator can perform teaching in the system, and the system records the teaching actions and generates corresponding optimization strategies.

[0195] Teaching data collection:

[0196] The user completes the target task through the teaching device or directly controls the robot. The system automatically records every action, position change, sensor feedback and other information of the robot during the teaching process. The teaching data is used as a supervision sample for reinforcement learning to generate the initial strategy for the robot task.

[0197] Supervised learning training:

[0198] The system uses the data from the teaching process for supervised learning. By inputting the teaching data, the system trains the model and generates an initial strategy. This process greatly shortens the robot's autonomous learning time, allowing the robot to quickly master new tasks.

[0199] Teaching data optimization and combination:

[0200] The teaching data is combined with the strategy generated by reinforcement learning to further optimize the action program. The system compares the execution effect of the teaching strategy and the reinforcement learning strategy, selects the action sequence with the best effect, and optimizes the strategy through model fusion.

[0201] 5. Sensor feedback and data processing implementation

[0202] During the execution of the task, the robot obtains real-time environmental information through a variety of sensors (such as vision, lidar, force sensors, etc.), and integrates and processes this data for dynamic adjustment during task execution.

[0203] Multi-sensor fusion:

[0204] The visual sensor obtains image information of the environment, the laser radar is used to detect the distance of objects around the robot, and the force sensor is used to sense the force during operation. The system integrates these sensor data to generate overall perception information of the environment.

[0205] Data preprocessing and state estimation:

[0206] The data collected by the sensor is de-noised, filtered and normalized to ensure the accuracy and consistency of the data. The system estimates the current state of the robot (such as position, posture, speed) through algorithms such as Kalman filtering to provide support for real-time adjustment of action strategies.

[0207] Feedback Mechanism:

[0208] The system monitors the robot's status information during task execution in real time, and combines sensor feedback to correct errors in task execution in real time. For example, when performing a grasping operation, if the force feedback perceived by the robot does not match expectations, the system will automatically adjust the action path and force to ensure that the task is completed smoothly.

[0209] 6. Controller and execution module implementation

[0210] The controller and execution module are responsible for controlling the robot to complete specific operation tasks according to the strategies generated by the autonomous programming engine.

[0211] Motion Controllers:

[0212] The motion controller is responsible for precisely controlling the motion of the robot's joints and end effectors, ensuring the accuracy and stability of the movements when performing tasks. Through the path planning algorithm, the robot can select the optimal path during task execution, reducing execution time and improving task efficiency.

[0213] Security control mechanism:

[0214] During the mission execution, the system has a safety control mechanism to prevent the robot from performing dangerous actions. For example, during the exploration process, if the system detects that the robot is about to collide, the system will automatically trigger a stop signal and adjust the action path to ensure the safety of the operation.

[0215] 7. Communication and synchronization implementation

[0216] The system realizes data interaction between modules through a high-speed communication network and ensures time synchronization in multi-robot collaborative tasks.

[0217] Real-time communication:

[0218] The system uses industrial Ethernet or wireless network to achieve high-speed data transmission between the robot and the control system. Through standardized communication protocols, seamless data interaction between modules is ensured, ensuring real-time performance during task execution.

[0219] Time Synchronization:

[0220] In multi-robot collaborative scenarios, the system achieves time synchronization through the Precision Time Protocol (PTP) to ensure that multiple robots perform coordinated actions when performing tasks together.

[0221] 8. Implementation of human-computer interaction layer

[0222] The human-computer interaction layer provides users with an operation interface, which facilitates users to set tasks, monitor task progress, conduct teaching and adjust system parameters.

[0223] Task Setting and Management:

[0224] Users input task instructions and manage task execution status through an intuitive graphical interface. The system converts high-level instructions into specific execution strategies, and users can monitor task progress and robot status at any time.

[0225] Monitoring and Visualization:

[0226] The system provides a visual interface for task execution, where users can view the robot’s current status, path, sensor feedback information, and key steps of task execution, and understand the progress of the task in real time.

[0227] Alarm and intervention:

[0228] During the execution of the task, if an abnormal situation occurs, the system will automatically issue an alarm prompt, and the user can manually intervene or adjust the system parameters through the interface to ensure the smooth completion of the task.

[0229] As an optional implementation, refer to Figure 2 , the operation process of the system of the present invention is:

[0230] 1. Task setting and programming:

[0231] The operator sets the task through the human-computer interaction interface, enters simplified high-level task instructions, and describes the task objectives (such as "grab an object and place it in a specified location"). The autonomous programming engine receives the task description and generates reinforcement learning tasks based on the basic actions in the behavior library.

[0232] 2. Simulated environment training:

[0233] In virtual simulation platforms (such as Gazebo, Webots, etc.), robots are initially trained for tasks using reinforcement learning algorithms. The system builds a virtual task environment and performs multiple training iterations. The robot gradually learns the strategy to complete the task through the loop feedback of state, action, and reward.

[0234] 3. Training results analysis and optimization:

[0235] After the training is completed, the system analyzes the training results and evaluates the effectiveness and efficiency of task completion. The system optimizes the strategy based on the reward function and records indicators such as the success rate and error rate of each round of training to form a preliminary strategy model.

[0236] 4. Real environment strategy migration:

[0237] The policy model obtained in the simulation environment is transferred to the real robot. In the real environment, the robot collects information about the surrounding environment through sensors, and the system uses transfer learning technology to fine-tune the policy to adapt it to the physical characteristics and environmental changes of the real world.

[0238] 5. Sensor feedback and real-time adjustment:

[0239] When performing tasks in a real environment, the robot obtains environmental and status feedback through visual sensors, lidar, force sensors, etc. The system processes this data in real time to ensure the smooth execution of the task and dynamically adjusts the task strategy based on the feedback.

[0240] 6. User teaching optimization:

[0241] The operator can manually guide the robot to complete the task through the teaching function, and the system records the action data during the teaching process. The system uses the teaching data for supervised learning, optimizes the robot strategy, and further improves the accuracy and efficiency of task execution.

[0242] 7. Task execution and collaboration:

[0243] The optimized robot strategy begins to perform tasks in the real environment. Multiple robots work together to execute specific action sequences through the controller module to ensure accurate completion of the task. In the multi-robot collaboration scenario, the communication module ensures the synchronization of the actions of each robot.

[0244] 8. Status feedback and continuous optimization:

[0245] During the task execution, the system continuously monitors the robot's status and task progress, and adjusts the execution strategy in real time through sensor feedback and task completion status. At the same time, the system continues to conduct online learning and optimize the strategy model to cope with future task changes.

[0246] 9. Result recording and evaluation:

[0247] After the task is completed, the system records the key data of the execution process, including task completion time, accuracy, error rate, etc. The system evaluates the results of task execution, identifies the parts that need improvement, and provides reference for subsequent task execution.

[0248] As an optional implementation, the present invention verifies the following technical solution of the robot autonomous programming system based on reinforcement learning:

[0249] 1. Experimental environment

[0250] The experimental environment of the present invention includes two parts: virtual simulation and real physical environment. In the virtual environment, a simulation platform (such as Gazebo or Webots) is used to create realistic virtual robot models and task scenarios, simulate the physical properties and task requirements in reality, and ensure the effectiveness of the training strategy. The virtual training server is equipped with high-performance computing resources (such as NVIDIAA100 GPU, 512GB memory, 2TB SSD storage) for acceleration and parallel training of reinforcement learning algorithms. In a real physical environment, the experiment uses a collaborative robot equipped with a variety of sensors (such as visual sensors, lidars, and force sensors), which is connected to the control server through a high-speed communication network, and ensures time synchronization and real-time data transmission in the execution of multi-robot tasks, verifying that the training results in the virtual environment can be migrated and executed in the actual scene.

[0251] 2. Experimental Procedure

[0252] Step 1: Experimental environment configuration

[0253] Virtual environment construction:

[0254] In the Gazebo or Webots simulation platform, build robot models and task scenarios to ensure that the physical properties of the simulation environment (such as friction, gravity, collision, etc.) are consistent with the real physical environment. Task scenarios include complex operations such as object grasping, handling, and assembly.

[0255] Hardware equipment preparation:

[0256] Configure the hardware required for the experiment, including a multi-sensor robot (equipped with visual sensors, lidar, force sensors, etc.) and a high-performance computing server (such as NVIDIA A100 GPU, 512GB memory). Build a high-speed communication network and ensure the stability of the industrial Ethernet or wireless network.

[0257] Reinforcement Learning Algorithm Deployment:

[0258] Deploy deep reinforcement learning algorithms (such as DDPG or PPO) in the server for task strategy training in the virtual environment. Configure a multi-threaded parallel training mechanism to ensure the efficiency of the training process.

[0259] Step 2: Virtual environment training

[0260] Initialize task training:

[0261] In a virtual environment, the robot is initially trained using a reinforcement learning algorithm. The training tasks include the robot completing basic operations such as grabbing and moving objects in a virtual scene. The system optimizes the robot's action strategy based on the state-action-reward feedback mechanism.

[0262] Strategy Optimization and Convergence:

[0263] The system optimizes the robot strategy through multiple rounds of training, adjusts the reward function based on indicators such as task completion time, success rate, error rate, and accelerates training convergence. Through experience replay and policy gradient update, the stability and efficiency of model training are improved.

[0264] Step 3: Real environment migration and fine-tuning

[0265] Strategy transfer to real robot:

[0266] The strategy trained in the virtual environment is transferred to the real robot. In the real environment, the robot obtains environmental feedback through sensors such as vision, force perception and lidar, and adjusts the strategy based on the sensor data.

[0267] Transfer learning and fine-tuning:

[0268] Through transfer learning technology, the system fine-tunes the strategy and adjusts the strategy deviation caused by the difference between the virtual environment and the real environment. Sensor data is input into the system as feedback information to help the robot adaptively adjust its action path and operation mode.

[0269] Step 4: User teaching optimization

[0270] Teaching guidance:

[0271] Through the teaching function, the operator manually guides the robot to perform tasks, and the system records the data during the operation (such as position, movement, force, etc.). The teaching process can accelerate the robot's adaptation to new tasks or more complex scenarios.

[0272] Teaching data application and optimization:

[0273] The system uses the teaching data for supervised learning and optimizes the teaching actions in combination with the reinforcement learning strategy. By combining with the reinforcement learning strategy, more efficient task execution actions are generated.

[0274] Step 5: Task execution and feedback

[0275] Task Execution:

[0276] The robot performs tasks in the real environment according to the optimized strategy and completes the target operation (such as object grasping and handling). The controller ensures the accuracy of the action and the stability of the task execution based on the real-time sensor feedback.

[0277] Feedback and Adjustment:

[0278] During the task execution, the system monitors the robot status in real time and adjusts the task strategy based on sensor feedback to ensure the successful completion of the task. The robot continuously learns and optimizes the action strategy to improve the efficiency of task execution.

[0279] Step 6: Record and analyze results

[0280] Data Records:

[0281] Record key data such as task completion time, success rate, error rate, etc. for each experiment. Collect information such as robot sensor feedback, execution path, task completion status, etc. as a reference for subsequent optimization.

[0282] Experimental analysis and evaluation:

[0283] By analyzing the experimental data, we evaluate the robot's task completion effect in virtual and real environments, and analyze the differences between virtual training and real execution. Based on the evaluation results, we further optimize the system performance and improve the training and execution plans.

[0284] The robot autonomous programming system based on reinforcement learning of the present invention has the following significant beneficial effects:

[0285] 1. Simplify the robot programming process and lower the programming threshold

[0286] Traditional robot programming usually requires specialized programming languages ​​and complex task logic design, which is highly professional and complex to operate. The present invention uses an autonomous programming engine, and users only need to operate through high-level task descriptions. The system automatically parses the tasks and generates corresponding execution strategies, which greatly reduces the dependence on manual programming and simplifies the robot operation process.

[0287] 2. Improve the autonomy and flexibility of robot task execution

[0288] Through reinforcement learning algorithms, robots can autonomously learn the best execution strategy for tasks in a virtual environment and flexibly adjust action sequences when faced with multiple tasks and different work scenarios. The system can dynamically generate action programs based on changes in tasks and environments, has extremely high adaptability, and is particularly suitable for complex and changeable industrial application scenarios.

[0289] 3. Improve task training and execution efficiency

[0290] The present invention uses parallel training and virtual simulation technology to greatly speed up the generation of robot task execution strategies. By having multiple robots perform task training in a virtual environment at the same time, the training cycle is significantly shortened. After the strategy is optimized in the virtual environment, it is further adjusted in the real environment through transfer learning, ensuring that the training results can be effectively applied to actual task scenarios, thereby improving the efficiency of task execution.

[0291] 4. Reduce the difference between virtual and real environments and optimize the migration effect

[0292] This invention effectively narrows the gap between the virtual environment and the real environment (Sim-to-Real Gap) through transfer learning technology, and fine-tunes the strategy in combination with sensor feedback to ensure accurate execution of the strategy in the real environment. This process greatly reduces the trial and error cost in real tasks and improves the execution stability of the robot in a variety of environments.

[0293] 5. Enhance the system’s usability and human-computer interaction

[0294] The system provides a user-friendly teaching optimization module, allowing operators to quickly guide the robot to complete new tasks through teaching. By collecting teaching data and combining supervised learning technology, the system can quickly generate or adjust task execution strategies, greatly improving the ease of use of the system and lowering the technical threshold for users. Operators do not need to have professional programming skills to effectively use the system.

[0295] 6. Improve the safety and reliability of task execution

[0296] Through multi-sensor fusion and real-time feedback mechanisms, the system can continuously obtain environmental information and make dynamic adjustments during the task execution to ensure the accuracy and safety of the robot operation. The controller module combines real-time feedback to dynamically adjust the task to prevent the robot from making misoperations during the task execution, ensuring the safety and efficiency of the task.

[0297] 7.Support the execution of various complex industrial tasks

[0298] The system of the present invention can adapt to a variety of industrial scene tasks, including object grasping, handling, assembly, etc. Through task training, teaching optimization and strategy adjustment, the system can flexibly adapt to different operation requirements. This multi-task adaptability makes the system have broad application prospects in the fields of industrial manufacturing, logistics handling, etc.

[0299] 8. Save labor costs and improve production efficiency

[0300] By autonomously learning to generate action programs and reducing reliance on manual programming, the system of the present invention effectively reduces reliance on highly professional personnel and reduces the need for manual intervention. Combined with efficient reinforcement learning and transfer learning technologies, the system can be quickly deployed in actual production environments, thereby improving overall production efficiency and reducing labor costs.

[0301] The key points of the present invention are:

[0302] 1. A robot autonomous programming system based on reinforcement learning of the present invention comprises the following modules:

[0303] An autonomous programming engine, which is used to generate task instructions based on the high-level task description input by the user, combined with the behavior library, and generate the optimal action program through reinforcement learning algorithm;

[0304] The behavior library is used to store the basic action units of the robot, including movement, grasping, rotation, placement and other action units, providing basic operations for task execution;

[0305] The virtual environment training module is used to train the robot on tasks through a reinforcement learning algorithm in a virtual simulation platform and generate a strategy model for completing the task;

[0306] The real environment transfer module is used to transfer the training strategy in the virtual environment to the real robot, and fine-tune the strategy by combining sensor feedback and transfer learning technology;

[0307] The user teaching optimization module is used to collect the operator's teaching data through the teaching device, and optimize the robot's action strategy for executing tasks in combination with supervised learning;

[0308] The sensor feedback and data processing module is used to fuse data from sensors such as visual sensors, force sensors, and lidar, perform preprocessing and state estimation, and is used for dynamic adjustment during task execution;

[0309] The controller and execution module is used to receive the strategy instructions from the autonomous programming engine and control the robot to perform task operations;

[0310] The communication and synchronization module is used to realize data transmission between system modules and time synchronization of multi-robot collaborative tasks.

[0311] 2. The autonomous programming engine parses the task description input by the user through a high-level task instruction set. The task instruction set is expressed in natural language or simplified task description statements and converted into corresponding action instructions.

[0312] 3. The action units in the behavior library can be dynamically combined according to different tasks and iteratively optimized during the reinforcement learning process to ensure the optimal action combination for each task.

[0313] 4. The virtual environment training module trains the robot through deep reinforcement learning algorithms (such as DDPG and PPO) combined with the state-action-reward mechanism, and updates the strategy model through multiple iterations to optimize the task execution effect.

[0314] 5. The virtual environment training module uses parallel training technology. Multiple robots are trained simultaneously in a virtual environment, which improves training efficiency and sample size, and accelerates model convergence through distributed strategies.

[0315] 6. The real environment transfer module reduces the difference between the simulated environment and the real environment (Sim-to-Real Gap) through transfer learning methods, and combines real-time sensor data to perform online fine-tuning of the strategy.

[0316] 7. The sensor feedback and data processing module integrates the feedback information of vision, lidar and force sensors, performs filtering, denoising and normalization to improve data accuracy, and estimates the state of the robot through Kalman filtering or particle filtering algorithm.

[0317] 8. The user teaching optimization module collects the operator's teaching actions through a teaching device (such as an operating handle or a physical guide) and converts them into action sequences for robot strategy optimization and supervised learning.

[0318] 9. The controller and execution module control the robot to perform tasks according to the action sequence generated by the strategy model, and dynamically adjust the execution actions through the real-time feedback mechanism to ensure the accuracy and efficiency of the task.

[0319] 10. The communication and synchronization module realizes data transmission between modules through industrial Ethernet or wireless communication to ensure the real-time execution of robot tasks, and realizes time synchronization of multi-robot tasks through precise time protocol (PTP) or other synchronization technologies.

[0320] 11. The system can improve the robot's adaptability in different tasks through dual optimization of user teaching and virtual environment, and supports adaptive adjustment of various complex tasks, including object grasping, handling, assembly and other industrial application scenarios.

[0321] 12. The task execution process in the virtual environment and the real environment is combined through transfer learning, and the robot strategy is adjusted in real time to cope with the changing task environment, ensuring that it still has efficient task execution capabilities in a dynamic task environment.

[0322] As an optional implementation, the teaching data application and optimization method of the present invention includes:

[0323] 1. Acquisition and processing of teaching data

[0324] Data collection: During the teaching stage, the operator controls the robot to perform tasks through teaching devices (such as handles, joysticks or physical guides), and the system collects key data in the teaching process in real time, including robot movements, positions, speeds, force information, etc. Environmental feedback is recorded synchronously through sensors (such as vision and force sensors) to ensure the accuracy and completeness of the teaching data.

[0325] Data preprocessing: Teaching data usually contains noise, so the data needs to be preprocessed before optimization. The specific steps include:

[0326] 1) Data denoising: Use filtering techniques (such as Kalman filtering or low-pass filtering) to remove noise from sensor data.

[0327] 2) Data normalization: Normalize different data dimensions to ensure that data is input into the model at a uniform scale, avoiding the impact of data scale differences on model training.

[0328] 3) Key frame extraction: Extract key frames from continuous actions during the teaching process to avoid redundant data and ensure that only action information that is critical to task execution is retained.

[0329] 2. Supervised Learning and Optimization Strategies

[0330] Supervised learning application: The system uses the processed teaching data for supervised learning to generate an initial action strategy. The goal of supervised learning is to learn the mapping relationship from the teaching data. The specific steps include:

[0331] Input layer: The teaching state (such as the robot's position, speed, and force information) is used as the input feature, combined with sensor feedback information.

[0332] Output layer: Taking the operator's actions during the teaching process as output, the system generates an action sequence through a supervised learning model (such as decision tree, SVM or neural network, etc.) to preliminarily simulate the operator's action intentions.

[0333] Error feedback: By comparing the model output with the actual action sequence taught, the error is calculated, and the model parameters are adjusted using optimization algorithms such as gradient descent to gradually reduce the error.

[0334] 3. Strategy optimization combined with reinforcement learning

[0335] Reinforcement learning strategy generation: After the system generates a preliminary strategy through supervised learning, it further optimizes the teaching data by combining the reinforcement learning algorithm. The specific optimization process is as follows:

[0336] State-Action-Reward Cycle: Through the interaction between the robot and the environment, the system selects actions based on the current state and obtains reward feedback after execution. The reward function evaluates the quality of the action, and the system optimizes the action selection strategy to ensure the efficiency and accuracy of task completion.

[0337] Experience replay mechanism: Using the experience replay mechanism, the system combines the teaching data with the exploration data in reinforcement learning, and continuously replays the teaching data to improve the stability of the strategy and the accuracy of task execution.

[0338] Strategy update: The system adjusts the strategy network based on reward feedback and optimizes the action sequence of the robot to perform tasks through deep reinforcement learning algorithms (such as DDPG, PPO, etc.) to ensure the generation of the optimal strategy.

[0339] 4. Data analysis and optimization steps

[0340] Step 1: Behavior library construction: The system combines the teaching data with the action strategy generated by reinforcement learning to build a robot behavior library. The behavior library stores the basic actions that the robot may use when performing tasks (such as grasping, moving, rotating, etc.), providing a basis for subsequent task optimization.

[0341] Step 2: Strategy evaluation and feedback: The strategy generated by executing the teaching data multiple times is evaluated to evaluate the task completion, including task success rate, execution time, energy consumption, etc. The system evaluates the effectiveness of the strategy through quantitative indicator feedback and calculates the reward value.

[0342] Step 3: Policy update: Based on the evaluation results, the policy model is adjusted using the gradient update algorithm to reduce errors. Through repeated iterations, the system gradually optimizes the action sequence to ensure more accurate policy generation.

[0343] Step 4: Online learning and fine-tuning: During the actual task execution, the system combines sensor data for real-time monitoring and fine-tuning to ensure the flexibility and safety of the strategy in task execution. Through the online learning mechanism, the system can continuously optimize according to environmental changes.

[0344] 5. Control method

[0345] Task decomposition and action sequence generation: The system decomposes complex tasks into multiple subtasks, each of which corresponds to a specific action sequence. Through the strategy generated by the reinforcement learning algorithm, the robot gradually executes these subtasks and finally completes the overall task.

[0346] Feedback control: The system monitors the robot's status (such as position, speed, force, etc.) in real time and adjusts the action parameters in execution based on sensor feedback. Feedback control ensures the stability and safety of task execution by quickly responding to adjustment strategies.

[0347] On the other hand, an embodiment of the present invention also provides a robot autonomous programming device based on reinforcement learning, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the above method when executing the computer program.

[0348] The robot autonomous programming device based on reinforcement learning in the embodiment of the present invention includes a memory and a processor.

[0349] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0350] The memory may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage. Among them, ROM can store static data or instructions required by the processor or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0351] The memory stores executable codes, and when the executable codes are processed by the processor, the processor can execute part or all of the methods described above.

[0352] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0353] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0354] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A robot autonomous programming system based on reinforcement learning, characterized in that: The system includes an autonomous programming engine module, a simulated environment training module, a real environment migration module, a sensor feedback and data processing module, a controller and an execution module; The autonomous programming engine module is used to convert high-level task instructions into low-level action programs; The simulation environment training module is used to provide a virtual training platform and reinforce learning training; The real environment migration module is used to migrate the strategy learned from the simulation environment to the real environment; The sensor feedback and data processing module is used to collect and process various sensor data of the robot; The controller and execution module are used to receive instructions from the autonomous programming engine module and control the robot to perform tasks.

2. The system according to claim 1, characterized in that The autonomous programming engine module includes a behavior library, a high-level task instruction set submodule, and a reinforcement learning strategy generation submodule; The behavior library is used to store basic actions performed by the robot, and the basic actions include moving, grasping, and rotating; The high-level task instruction set submodule is used to simplify the instruction set input by the user; The reinforcement learning strategy generation submodule is used to automatically learn the task strategy through the reinforcement learning algorithm to generate an optimized action sequence.

3. The system according to claim 1, characterized in that The simulation environment training module includes a simulation platform, a reinforcement learning training submodule, and a virtual environment optimization submodule; The simulation platform is used to simulate the interaction between the robot and the physical environment and create a task scenario; The reinforcement learning training submodule is used to optimize the robot task execution strategy using the reinforcement learning algorithm and adjust the robot behavior through the reward feedback mechanism; The virtual environment optimization submodule is used to enhance the diversity of the simulation environment through domain randomization technology.

4. The system according to claim 1, characterized in that The real environment migration module includes a sensor data acquisition submodule, a migration learning submodule, and a strategy fine-tuning submodule; The sensor data acquisition submodule is used to obtain online environmental information through sensors; the sensors include visual sensors, force sensors, and laser radars; The transfer learning submodule is used to adjust the strategy obtained in the simulation environment through the transfer learning technology; The strategy fine-tuning submodule is used to perform adjustments through online training in a real environment.

5. The system according to claim 1, characterized in that The sensor feedback and data processing module includes a multi-sensor fusion submodule, a data preprocessing submodule, and a state estimation submodule; The multi-sensor fusion submodule is used to integrate data from sensors to form a comprehensive perception of the environment; The data preprocessing submodule is used to perform denoising, filtering and standardization on the sensor data; The state estimation submodule is used to estimate the current position and speed state of the robot through the Kalman filter algorithm.

6. The system according to claim 1, characterized in that The controller and execution module includes a motion controller submodule, a path planning and execution submodule, and a safety control mechanism submodule; The motion controller submodule is used to control the motion of the robot; The path planning and execution submodule is used to perform path planning according to the learned strategy; The safety control mechanism submodule is used to control the robot to avoid collision during mission execution.

7. The system according to claim 1, characterized in that The system also includes a user teaching optimization module and a human-machine interaction layer module; the user teaching optimization module is used to collect the teaching data of the operator and optimize the action strategy of the robot to perform the task in combination with supervised learning; The user teaching optimization module includes a teaching data recording submodule, a supervised learning submodule, and a user-friendly interface submodule; the human-computer interaction layer module is used to provide an operation interface; The teaching data recording submodule is used to record the action data when the operator guides the robot through manual operation or teaching equipment; The supervised learning submodule is used to use the teaching data for supervised learning to generate a strategy for the robot; The user-friendly interface submodule is used to provide a user interface; The human-computer interaction layer module includes a visual monitoring interface submodule, a task setting and modification submodule, and an alarm and prompt module submodule; The visual monitoring interface submodule is used to display robot status, task progress, and sensor data information; The task setting and modification submodule is used to define task objectives and execution parameters; The alarm and prompt module submodule is used to issue an alarm when an abnormality occurs in a task.

8. The system according to claim 1, characterized in that The system also includes a communication and synchronization module; the communication and synchronization module is used to coordinate the cooperation of multiple robots; The communication and synchronization module includes a high-speed communication network submodule and a time synchronization mechanism submodule; The high-speed communication network submodule is used to realize online data transmission between various devices through industrial Ethernet or wireless network; The time synchronization mechanism submodule is used to perform motion synchronization when multiple robots collaborate through a precise time protocol.

9. A robot autonomous programming method based on reinforcement learning, characterized in that: The method comprises the following steps: Through the autonomous programming engine module, high-level task instructions are converted into low-level action programs; Through the simulation environment training module, a virtual training platform and reinforcement learning training are provided; Through the real environment transfer module, the strategies learned from the simulation environment are transferred to the real environment; Collect and process various sensor data of the robot through the sensor feedback and data processing module; Through the controller and the execution module, the instructions from the autonomous programming engine module are received, and the robot is controlled to perform tasks.

10. The method according to claim 9, characterized in that The method further comprises the following steps: Get task information; Constructing a virtual task environment for training according to the task information, and obtaining training result information; Perform transfer learning based on the training result information to obtain task strategy transfer information; Acquire mission strategy optimization information according to the mission strategy migration information and sensor collection data; According to the task strategy optimization information, the robot autonomous programming is completed.

Images