Robot real machine reinforcement learning method and device, storage medium and program product

Abstract

The invention provides a robot real machine reinforcement learning method and device, a storage medium and a program product, and relates to the technical field of robots, and the method comprises the steps: obtaining environment visual information when a robot executes an interaction task in a real physical environment; performing image background enhancement processing on the environment visual information to obtain background-enhanced target environment visual information; and constructing environment interaction experience data based on the target environment visual information, and performing real machine reinforcement learning on the robot based on the environment interaction experience data. According to the invention, the problem of insufficient background generalization of robot real machine reinforcement learning can be solved.

Description

Technical Field

[0001] This application relates to the field of robot operation technology, and in particular to a robot real machine reinforcement learning method, electronic device, storage medium and program product. Background Technology

[0002] The mainstream robot learning methods in the field of robot manipulation mainly include imitation learning and reinforcement learning. Reinforcement learning methods are further divided into two categories: simulation-based reinforcement learning and real-machine reinforcement learning, which involves training directly on a real machine. Simulation-based reinforcement learning involves rapid training in a simulation environment followed by transfer to a real machine. Real-machine reinforcement learning, on the other hand, is trained directly on a real machine, thus eliminating the gap between simulation and real-machine training. However, since real-machine reinforcement learning generally uses visual information as input, while the introduction of visual information brings rich state information, it also introduces the problem of insufficient background generalization. Summary of the Invention

[0003] The main purpose of this application is to propose a robot real-machine reinforcement learning method, device, storage medium, and program product, which aims to solve the problem of insufficient background generalization in robot real-machine reinforcement learning.

[0004] To achieve the above objectives, the first aspect of this application proposes a real-machine reinforcement learning method for robots, the method comprising: Acquire environmental visual information when the robot performs interactive tasks in a real physical environment; The environmental visual information is subjected to image background enhancement processing to obtain the target environmental visual information with enhanced background. Environmental interaction experience data is constructed based on the visual information of the target environment, and the robot is subjected to real-machine reinforcement learning based on the environmental interaction experience data.

[0005] In some embodiments, the target environment visual information is obtained by image background enhancement processing on the environment visual information in the first environment state observation data, and the first environment state observation data is the environment state observation data of the robot when responding to human control in the real physical environment. The construction of environmental interaction experience data based on the visual information of the target environment includes: The reward model is trained based on the expert interaction action data of the robot in response to human control in the real physical environment and the visual information of the target environment to obtain the trained reward model. In response to the robot interacting with the real physical environment, dense reward data for the robot is generated based on the trained reward model; The robot's environmental interaction experience data is constructed based on the dense reward data.

[0006] In some embodiments, generating dense reward data for the robot based on the trained reward model includes: Acquire the autonomous interaction action data of the robot; the autonomous interaction action data is the interaction action data generated by the robot's reinforcement learning model when the robot interacts with the real physical environment; Based on the trained reward model, the autonomous interaction action data and the corresponding second environmental state observation data are processed to obtain the robot's dense reward data; the environmental state observation data includes at least the environmental visual information when the robot interacts with the real physical environment.

[0007] In some embodiments, constructing the robot's environmental interaction experience data based on the dense reward data includes: An autonomous interaction action data sequence is generated based on the autonomous interaction action data, the second environmental state observation data, and the dense reward data; the environmental interaction experience data includes the interaction action data sequence.

[0008] In some embodiments, the method further includes: Based on the expert interaction action data of the robot responding to human control and interacting with the real physical environment, the trained reward model is optimized.

[0009] In some embodiments, optimizing the trained reward model based on expert interaction action data when the robot responds to human control and interacts with the real physical environment includes: Based on the expert interaction action data of the robot, the third environmental state observation data corresponding to the expert interaction action data, and the rule reward information, an expert interaction action data sequence is generated. Reward experience data is constructed based on the expert interaction action data sequence, and the trained reward model is optimized based on the reward experience data.

[0010] In some embodiments, the image background enhancement processing of the environmental visual information includes: Randomly extract a portion of the environmental image data to be processed from the environmental visual information; The environmental image data is subjected to background segmentation processing, and the segmented background image data is subjected to image data enhancement processing to obtain background-enhanced target environment image data; the target environment visual information includes the target environment image data.

[0011] To achieve the above objectives, a second aspect of this application proposes a robot real-machine reinforcement learning device, the device comprising: The acquisition module is used to acquire environmental visual information when the robot performs interactive tasks in a real physical environment; The background enhancement module is used to perform image background enhancement processing on the environmental visual information to obtain the target environmental visual information with enhanced background. The real-machine reinforcement learning module is used to construct environmental interaction experience data based on the visual information of the target environment, and to perform real-machine reinforcement learning on the robot based on the environmental interaction experience data.

[0012] To achieve the above objectives, a third aspect of this application proposes an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the robot real-machine reinforcement learning method described in the first aspect.

[0013] To achieve the above objectives, a fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the robot real-machine reinforcement learning method described in the first aspect.

[0014] To achieve the above objectives, the fifth aspect of this application proposes a computer program product, which includes a computer program that, when executed by a processor, implements the robot real-machine reinforcement learning method provided in the first aspect above.

[0015] The robot real-machine reinforcement learning method, device, electronic device, computer-readable storage medium and computer program product proposed in this application acquire environmental visual information when the robot performs interactive tasks in a real physical environment, then perform image background enhancement processing on the environmental visual information to obtain background-enhanced target environmental visual information, then construct environmental interaction experience data based on the target environmental visual information, and then perform real-machine reinforcement learning on the robot based on the environmental interaction experience data.

[0016] Thus, this application innovatively introduces real-time image detection and segmentation technology into the real-machine reinforcement learning training process of robots. Based on data augmentation methods, the environmental visual information of the robot when performing interactive tasks in real physical environments is enhanced. Then, the environmental interaction experience data is constructed using the enhanced target environmental visual information to perform real-machine reinforcement learning on the robot. This can effectively solve the problem of insufficient background generalization ability in real-machine reinforcement learning, improve the robot's background generalization ability, and enable the robot to quickly adapt to different background scenarios. Attached Figure Description

[0017] Figure 1 A flowchart illustrating the steps of the robot real-machine reinforcement learning method provided in this application in some embodiments; Figure 2 A schematic diagram of the probabilistic image data augmentation process involved in some embodiments of the robot real-machine reinforcement learning method provided in this application; Figure 3 A schematic diagram of the expert trajectory data acquisition process involved in some embodiments of the robot real-machine reinforcement learning method provided in this application; Figure 4 for Figure 1 A detailed flowchart of step S103; Figure 5 The robot real-machine reinforcement learning method provided in this application is illustrated in some embodiments with a schematic diagram of the reinforcement learning process involved. Figure 6 for Figure 4 A detailed flowchart of step S302; Figure 7 A flowchart illustrating the steps of the robot real-machine reinforcement learning method provided in this application in some other embodiments; Figure 8 for Figure 7 A detailed flowchart of step S701; Figure 9 A schematic diagram of the overall process involved in some embodiments of the robot real-machine reinforcement learning method provided in this application; Figure 10 A schematic diagram of reinforcement learning model deployment scenarios involved in some embodiments of the robot real-machine reinforcement learning method provided in this application; Figure 11 A schematic diagram of the structure of an embodiment of the robot real-machine reinforcement learning device provided in this application; Figure 12 A schematic diagram of the hardware structure of an embodiment of the electronic device provided in this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] It should be noted that although functional modules are divided in the device / system schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device / system or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this application.

[0021] First, the overall concept of the robot real-machine reinforcement learning method provided in this application will be explained.

[0022] Robot operation is one of the core capabilities of intelligent robots, involving the close coupling of perception, planning and control.

[0023] With the development of service robots, industrial automation, and embodied intelligence, robots not only need to complete pre-programmed tasks but also need to have the ability to autonomously perform operational tasks in complex and dynamic environments. However, traditional model-based control and planning methods often struggle to complete tasks in situations with complex contact dynamics (such as grasping, plugging, and assembly), high visual perception noise, large environmental changes, and diverse task requirements. Therefore, learning-based robot manipulation methods have gradually become a research hotspot in recent years.

[0024] Current mainstream robot learning methods mainly include imitation learning and reinforcement learning. Imitation learning learns strategies through expert demonstrations, resulting in fast learning speed and high stability. However, imitation learning lacks exploratory capabilities and fails when the environment deviates from the data distribution, exhibiting poor generalization. Reinforcement learning, on the other hand, interacts with the environment and optimizes strategies based on reward signals. It does not rely on expert data, can discover new strategies, and has stronger generalization capabilities.

[0025] Currently, reinforcement learning methods in the field of robot manipulation fall into two categories: training in simulation environments and training on real machines. Training in simulation environments allows for easy acquisition of arbitrary states and design of dense rewards (e.g., reward design based on the distance to the target object), enabling rapid training across multiple parallel environments. However, the biggest challenge is the difficulty in transferring from simulation to real machines, especially in complex scenarios (rich in contact, flexibility, fluidity, and friction). Real-machine reinforcement learning, trained directly on a real machine, avoids the gap between simulation and real-machine learning, but suffers from low data efficiency, exploration risks, and sparse and difficult-to-design rewards. To address these issues, the field of robot manipulation has proposed a human-in-the-loop (HIL) real-machine reinforcement learning solution. However, since the pose and contact states of objects are difficult to acquire in real environments, traditional reinforcement learning reward design schemes based on simulation environments cannot be used in real-machine reinforcement learning. Furthermore, while the introduction of visual information brings rich state information, it also introduces the problem of insufficient background generalization in real-machine reinforcement learning, which is generally based on visual information as input.

[0026] To address the problem of insufficient generalization ability of robot reinforcement learning on real machines, this application provides a robot reinforcement learning method, device, electronic device, computer-readable storage medium, and computer program product. It acquires environmental visual information when the robot performs interactive tasks in a real physical environment, then performs image background enhancement processing on the environmental visual information to obtain background-enhanced target environmental visual information, then constructs environmental interaction experience data based on the target environmental visual information, and performs real machine reinforcement learning on the robot based on the environmental interaction experience data.

[0027] Thus, this application innovatively introduces real-time image detection and segmentation technology into the real-machine reinforcement learning training process of robots. Based on data augmentation methods, the environmental visual information of the robot when performing interactive tasks in real physical environments is enhanced. Then, the environmental interaction experience data is constructed using the enhanced target environmental visual information to perform real-machine reinforcement learning on the robot. This can effectively solve the problem of insufficient background generalization ability in real-machine reinforcement learning, improve the robot's background generalization ability, and enable the robot to quickly adapt to different background scenarios.

[0028] Furthermore, in related technologies, real-machine reinforcement learning reward schemes for robots generally employ sparse rewards based on final success or failure. However, sparse rewards suffer from slow reward propagation, slow training speed, and difficulty in completing long-term tasks. Other real-machine reinforcement learning methods may use QR codes to obtain the target object's pose and then design dense rewards based on distance. However, this method requires affixing and calibrating QR code markers to the target object, a complex process. This method is generally only suitable for laboratory use and difficult to implement in real-world scenarios. Moreover, some real-machine reinforcement learning methods estimate the target object's pose using 6D pose estimation methods from computer vision and then design dense rewards based on distance. This method also relies on camera calibration, and 6D pose estimation has a large error, making it difficult to handle symmetrical objects and occluded scenes.

[0029] To address these issues, this application provides a robot real-machine reinforcement learning method, apparatus, electronic device, computer-readable storage medium, and computer program product. It also offers a reward design scheme that relies on expert action data and robot state data to train a deep learning model to generate rewards. This scheme not only provides dense rewards throughout the training process but also eliminates the need for additional markers and avoids the problems of inaccurate 6D pose estimation and calibration. Human intervention in the action training of the reward model allows for convenient embedding of dense rewards into the human-in-the-loop robot real-machine reinforcement learning training process.

[0030] Next, the robot real-machine reinforcement learning method, apparatus, electronic device, computer-readable storage medium and computer program product provided in this application will be specifically described through the following embodiments, and firstly, the various detailed embodiments of the robot real-machine reinforcement learning method provided in this application will be described in detail.

[0031] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0032] It should be noted that the robot real-machine reinforcement learning method provided in this application relates to the field of embodied intelligence technology. The robot real-machine reinforcement learning method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a terminal configured on a robot, such as a robotic arm, or an electronic device associated with the robot, such as a smartphone, tablet, laptop, or desktop computer. The association between the terminal and the robot means that the terminal can communicate and interact with the robot via a network. The server can be the robot's backend server terminal device, which can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The software can be an application implementing the robot real-machine reinforcement learning method, a computer program, and a storage medium carrying the computer program. It should be understood that, based on different design needs of practical applications, the terminal, server, and software of the robot real-machine reinforcement learning method provided in this application may also be other forms not listed here, depending on the different feasible embodiments. The robot real-machine reinforcement learning method provided in this application does not specifically limit these.

[0033] Furthermore, this application can also be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: robots such as robotic arms, personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, personal computers (PCs), minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via communication networks. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0034] For ease of understanding and explanation, the following text will use the application of the robot reinforcement learning method provided in this application to a terminal device as an example to describe the various specific embodiments of this application in detail. The terminal device can use the robot reinforcement learning method provided in this application to control the robot, or the terminal device can be integrated into or associated with the robot to manage the robot's task planning and execution processes. In some descriptions, the terminal device may be simply referred to as a terminal. The implementation of the robot reinforcement learning method provided in this application using any of the above-described forms of subject matter can refer to the process of applying the robot reinforcement learning method to a terminal device described below.

[0035] Please refer to Figure 1 , Figure 1 The flowchart illustrates the steps of the robot real-machine reinforcement learning method provided in this application in some embodiments. It should be understood that, although... Figure 1 The flowcharts illustrating subsequent steps show the execution order of some method steps. However, based on different design needs in practical applications, the robot reinforcement learning method provided in this application can, of course, employ a different execution order of method steps than shown in the figures. That is, Figure 1 The order of the steps shown does not constitute a limitation on the execution logic order of the robot real-machine reinforcement learning method provided in this application. Any other methods based on this principle are not limited by the provided order. Figure 1 Reasonable changes to the sequence of steps shown should be included within the protection scope of the robot real-machine reinforcement learning method provided in this application.

[0036] like Figure 1 As shown, in some embodiments, the application of the robot real machine reinforcement learning method provided in this application by the terminal device may include steps S101 to S103 as shown below.

[0037] Step S101: Obtain environmental visual information when the robot performs interactive tasks in a real physical environment.

[0038] It should be noted that interactive tasks can be interactions between the robot and the environment and / or interactions between the robot and target objects in the environment (such as workpieces placed on a lathe, water cups placed on a table, etc.).

[0039] The terminal device can control the robot to perform real-world reinforcement learning training in a real physical environment. This allows the robot to acquire environmental state observation data through multimodal sensing sensors when performing interactive tasks in a real physical environment. This environmental state observation data includes at least environmental visual information. For example, when the robot is a robotic arm, the environmental visual information can be visual image information acquired by a fixed-view camera in space and a camera at the end effector of the robotic arm.

[0040] In some embodiments, environmental state observation data may also include force and torque signals during the contact process between the robot and target objects in the environment, as well as information such as the joint states of the robot body.

[0041] Step S102: Perform image background enhancement processing on the environmental visual information to obtain the target environmental visual information with enhanced background.

[0042] After acquiring the environmental visual information of the robot performing interactive tasks in a real physical environment, the terminal device further performs image background enhancement processing on the environmental visual information to obtain the target environmental visual information with enhanced background.

[0043] In some embodiments, when performing image background enhancement processing on environmental visual information, the terminal device can extract foreground objects from the environmental visual information using a detection model or a segmentation model, and then randomize the background in the environmental visual information to output a background-randomized image. This background-randomized image is the target environmental visual information after background enhancement.

[0044] In some embodiments, the detection model can be implemented using a real-time small model such as an object detection algorithm (e.g., YOLO), or it can be implemented using a zero-shot segmentation model such as a Segment Anything Model (SAM) series. It should be understood that, based on different design needs of practical applications, different detection or segmentation models can be used by the terminal device in different feasible implementations to perform image background enhancement processing on environmental visual information. Furthermore, the robot reinforcement learning method provided in the embodiments does not limit the specific type and implementation method of the detection or segmentation model.

[0045] In some embodiments, when a terminal device performs image background enhancement processing on environmental visual information, the background randomization processing method may be to convert the background into Gaussian noise, or to divide the original image into blocks and then randomly combine them. It should be understood that, based on different design needs of practical applications, the terminal device can, of course, employ different background enhancement methods to perform image background enhancement processing on environmental visual information in different feasible implementations. The robot real-machine reinforcement learning method provided in this application does not limit the specific types and implementation methods of background randomization processing.

[0046] In some embodiments, the step of "performing image background enhancement processing on the environmental visual information" in step S102 above may include the following steps: Randomly extract a portion of the environmental image data to be processed from the environmental visual information; The environmental image data is subjected to background segmentation processing, and the segmented background image data is subjected to image data enhancement processing to obtain background-enhanced target environment image data; the target environment visual information includes the target environment image data.

[0047] When performing image background enhancement processing on environmental visual information, the terminal device can select a portion of the environmental image data for background randomization according to a certain probability. That is, the terminal device first randomly extracts a portion of the environmental image data to be processed from the environmental visual information according to a certain probability. Then, the terminal device performs background segmentation processing on the extracted environmental image data and performs image data enhancement processing on the segmented background image data, thereby obtaining the background-enhanced target environmental image data. This background-enhanced target environmental image data is a portion of the background-enhanced target environmental visual information (the other portion is the original image that was not extracted for background enhancement).

[0048] In some embodiments, the terminal device can perform image background enhancement processing on environmental visual information through a probabilistic image data enhancement module. The probabilistic image data enhancement module is the core module for achieving background generalization. Before the environmental state observation data collected by the terminal device during the robot's interactive tasks is placed into the data pool, it must be processed by this probabilistic image data enhancement module. The processing method involves extracting foreground objects using a detection model or segmentation model, then randomizing the background to output a background-randomized image.

[0049] For example, such as Figure 2 As shown, after the terminal device inputs the environmental state observation data into the probabilistic image data enhancement module, the module does not perform image background randomization on every piece of environmental visual information. Instead, it extracts a portion of the environmental image data with a certain probability p (e.g., p=0.5) for background randomization, and then outputs the enhanced target environment image and the original image with a probability of 1-p. This achieves better background randomization results.

[0050] Step S103: Construct environmental interaction experience data based on the visual information of the target environment, and perform real-machine reinforcement learning on the robot based on the environmental interaction experience data.

[0051] After acquiring the enhanced visual information of the target environment, the terminal device generates a corresponding robot motion data sequence based on this information. This sequence is then stored as unified experience data to construct environmental interaction experience data for real-world reinforcement learning of the robot. The terminal device can then use this environmental interaction experience data to further train the robot in a real physical environment.

[0052] In some embodiments, the terminal device can train a reward model based on visual information of the target environment, and then use the reward model to perform human-in-the-loop reinforcement learning on the robot.

[0053] In this embodiment, when the robot performs interactive tasks in a real physical environment, the terminal device acquires environmental state observation data of the robot through multimodal perception sensors. This environmental state observation data includes at least environmental visual information. Subsequently, the terminal device further performs image background enhancement processing on this environmental visual information to obtain enhanced target environmental visual information. Based on this target environmental visual information, the terminal device generates a corresponding robot action data sequence, and then performs unified experience storage on this robot action data sequence to construct environmental interaction experience data for real-machine reinforcement learning of the robot. In this way, the terminal device can further train the robot in a real physical environment using this environmental interaction experience data.

[0054] Thus, this application innovatively introduces real-time image detection and segmentation technology into the real-machine reinforcement learning training process of robots. Based on data augmentation methods, the environmental visual information of the robot when performing interactive tasks in real physical environments is enhanced. Then, the environmental interaction experience data is constructed using the enhanced target environmental visual information to perform real-machine reinforcement learning on the robot. This can effectively solve the problem of insufficient background generalization ability in real-machine reinforcement learning, improve the robot's background generalization ability, and enable the robot to quickly adapt to different background scenarios.

[0055] In some embodiments, the target environment visual information is obtained by image background enhancement processing on the environment visual information in the first environment state observation data, where the first environment state observation data is the environment state observation data of the robot responding to human control in the real physical environment.

[0056] like Figure 3 As shown, when the terminal device acquires environmental visual information of the robot performing interactive tasks in a real physical environment, it can do so during the expert trajectory data acquisition process, while humans control the robot to perform interactive tasks in a real physical environment through external devices (game controllers or spatial mice, etc.) (based on human intervention actions). Interact with the environment and perform random actions. (It will not be used for robot control, but participates in reward rule calculation.) When the robot acquires first environmental state observation data s through sensors and rule reward information r through the reward rule, the first environmental state observation data includes environmental visual information and robot body perception information (such as joint velocities). Then, the terminal device performs image background enhancement processing on the environmental visual information in the first environmental state observation data using the above step S102 and its refinement steps to obtain the background-enhanced target environmental visual information.

[0057] It should be noted that the reward rules mainly include two parts: expert action rewards and task completion rewards. For example, the calculation formula for the reward rules can be as follows: .

[0058] in, It is an action vector. It is the expert action vector. This represents the distance between the action vector and the expert action vector. It is the expert action reward coefficient, for example, 0.5. It's a switch signal that controls whether the operation was successful. If a success signal is received, then... Set to 1. If no success signal is received, then... It is 0. It is the task completion reward coefficient, for example, 10.

[0059] Please refer to Figure 4 , Figure 4 for Figure 1 A detailed flowchart of step S103.

[0060] like Figure 4 As shown, in some embodiments, the step of “constructing environmental interaction experience data based on the target environment visual information” in step S103 above may include steps S401 to S403 as shown below.

[0061] Step S401: Based on the expert interaction action data of the robot responding to human control in the real physical environment and the visual information of the target environment, the reward model is trained to obtain the trained reward model.

[0062] After the terminal device performs image background enhancement processing on the environmental visual information in the first environmental state observation data to obtain the target environmental visual information with enhanced background, it further uses the target environmental visual information and the expert interaction action data (i.e. human intervention actions) of the robot responding to human control in the real physical environment when the terminal device collects the target environmental visual information to train the reward model and obtain the trained reward model.

[0063] In some embodiments, after collecting expert trajectory data, the terminal device can train the reward model R(s,a) using a supervised learning method. Here, the model's input consists of the first environmental state observation information s (including visual information of the target environment after background enhancement) and the expert interaction action data a, and the output is the reward. .

[0064] In some embodiments, the reward model may employ a Multilayer Perceptron (MLP) model, and the loss function of this MLP model may be the MSE loss. For example, the formula for the loss function may be as follows: .

[0065] in, This is the model output, and r is the rule reward collected during the previous expert trajectory acquisition phase.

[0066] Once the reward model is trained, it can serve as a dense source of rewards for subsequent reinforcement learning training of the robot on a real machine.

[0067] Step S402: In response to the robot interacting with the real physical environment, generate dense reward data for the robot based on the trained reward model.

[0068] It should be noted that, as Figure 5 As shown, the human-in-the-loop reinforcement learning process for robots mainly includes an online interaction process and a learning process. The primary task of the online interaction process is to acquire interaction data between the robot and the real physical environment. Unlike the stage of collecting expert trajectory data, the reward generation rule in this online interaction process is changed from generating rewards based on existing reward rules to generating rewards jointly based on both existing reward rules and the trained reward model. The learning process is responsible for retrieving data sequences from the data cache to learn model parameters, and then sending these model parameters back to the online interaction process at fixed intervals to update the model parameters.

[0069] After obtaining the trained reward model, the terminal device processes the interaction data in the online interaction process based on the trained reward model when training the robot in real-machine reinforcement learning with humans in the loop, in response to the robot's interaction with the real physical environment, thereby generating dense reward data for the robot.

[0070] In some embodiments, during the online interaction process, if the robot's interaction with the real physical environment involves human intervention, a reward is generated by the reward rules, along with a random action and its corresponding reward. If no human intervention is involved, the reinforcement learning model generates interaction action data to interact with the real physical environment, and a reward model provides a reward; this reward is called a dense reward.

[0071] Please refer to Figure 6 , Figure 6 for Figure 4 A detailed flowchart of step S402.

[0072] like Figure 6 As shown, in some embodiments, the step of "generating dense reward data for the robot based on the trained reward model" in step S402 above may include steps S601 and S602 as shown below.

[0073] Step S601: Obtain the autonomous interaction action data of the robot; the autonomous interaction action data is the interaction action data generated by the robot's reinforcement learning model when the robot interacts with the real physical environment.

[0074] During the online interaction process of the terminal device, when the robot interacts with the real physical environment based on the interaction action data generated by the reinforcement learning model, the interaction action data is acquired as the robot's autonomous interaction action data.

[0075] Step S602: Based on the trained reward model, process the autonomous interaction action data and the second environmental state observation data corresponding to the autonomous interaction action data to obtain the robot's dense reward data; the environmental state observation data includes at least the environmental visual information when the robot interacts with the real physical environment.

[0076] It should be noted that the second environmental state observation data corresponding to the autonomous interactive action data can be the environmental state observation data of the robot obtained by the robot through multimodal perception sensors when the robot performs interactive tasks with the real physical environment using autonomous interactive action data.

[0077] After the terminal device acquires the robot's autonomous interaction action data, it further processes the autonomous interaction action data and the corresponding second environmental state observation data to calculate and output the robot's dense reward data.

[0078] Step S403: Construct the robot's environmental interaction experience data based on the dense reward data.

[0079] When the terminal device performs human-in-the-loop reinforcement learning training on the robot, after generating dense reward data for the robot based on the trained reward model, it further constructs a sequence of action data for the robot's interaction with the environment based on this dense reward data. This sequence of action data is then placed in a data cache to construct the robot's environmental interaction experience data. In this way, during the learning process, the data sequence can be retrieved from the data cache to learn model parameters. Subsequently, the model parameters are sent to the online interaction thread every fixed number of steps to update the model parameters, thereby achieving human-in-the-loop reinforcement learning training for the robot.

[0080] In some embodiments, step S403 described above may include the following steps: An autonomous interaction action data sequence is generated based on the autonomous interaction action data, the second environmental state observation data, and the dense reward data; the environmental interaction experience data includes the interaction action data sequence.

[0081] When the terminal device constructs the action data sequence of robot interaction with the environment based on dense reward data, it can generate an autonomous interaction action data sequence (s,a,r) from the robot's autonomous interaction action data a, the second environmental state observation data s, and the dense reward data r. This autonomous interaction action data sequence can then be used as the data sequence generated by the robot based on the policy (policy generation data) and placed into the reinforcement learning data cache (specifically, into the policy generation data cache).

[0082] In some embodiments, during the online interactive process of the robot performing real-machine reinforcement learning, the terminal device can also collect expert interaction action data and third environment state observation data when the robot responds to human control and interacts with the real physical environment. The reward information is generated by the reward rules mentioned above. Then, the expert interaction action data a, the third environment state observation data s, and the rule reward information r are used to generate an expert interaction action data sequence (s, a, r). This expert interaction action data sequence can be put into the reinforcement learning data cache (specifically into the expert intervention data) as a human intervention action data sequence.

[0083] It should be noted that the third environmental state observation data can be the environmental state observation data of the robot obtained by the robot through multimodal perception sensors when the terminal device performs interactive tasks with the robot using expert interactive action data and the real physical environment.

[0084] In this embodiment, the reward model is trained using a terminal device based on expert interaction action data of the robot responding to human control in a real physical environment, and observation data of the first environmental state including visual information of the target environment. This results in a trained reward model. Then, in response to the robot's interaction with the real physical environment, dense reward data for the robot is generated based on this trained reward model. This dense reward data is then used to construct environmental interaction experience data for real-machine reinforcement learning training of the robot. Thus, this embodiment innovatively introduces a human expert action reward model into the robot training process, realizing a dense-reward robot real-machine reinforcement learning method. This accelerates the reward transfer in robot real-machine reinforcement learning, speeds up the training process, and allows for faster task learning, effectively solving the problem of slow learning speed in sparse-reward robot real-machine reinforcement learning.

[0085] Please refer to Figure 7 , Figure 7 The flowcharts of the robot real-machine reinforcement learning method provided in this application are shown in some other embodiments.

[0086] like Figure 7 As shown, in some embodiments, the robot real machine reinforcement learning method provided in this application may further include the following step S701.

[0087] Step S701: Based on the expert interaction action data when the robot responds to human control and interacts with the real physical environment, optimize the trained reward model.

[0088] When the robot is undergoing real-machine reinforcement learning, the terminal device can also collect expert interaction action data when the robot responds to human control and interacts with the real physical environment during the online interaction process, and further optimize the reward model after training based on the expert interaction action data.

[0089] In some embodiments, the terminal device may construct reward experience data based on expert interaction action data, and then use the reward experience data to optimize the trained reward model.

[0090] Please refer to Figure 8 , Figure 8 for Figure 7 A detailed flowchart of step S701.

[0091] like Figure 8 As shown, in some embodiments, step S701 above: optimizing the trained reward model based on expert interaction action data when the robot responds to human control and interacts with the real physical environment, may include steps S801 and S802 as shown below.

[0092] Step S801: Generate an expert interaction action data sequence based on the robot's expert interaction action data, the third environmental state observation data corresponding to the expert interaction action data, and the rule reward information.

[0093] During the online interactive process of robot reinforcement learning, the terminal device can also collect expert interaction action data and third environment state observation data when the robot responds to human control and interacts with the real physical environment. The reward information is generated by the above-mentioned reward rules. Then, the expert interaction action data a, the third environment state observation data s, and the rule reward information r are used to generate an expert interaction action data sequence (s,a,r).

[0094] Step S802: Construct reward experience data based on the expert interaction action data sequence, and optimize the trained reward model based on the reward experience data.

[0095] After generating expert interaction action data sequences, the terminal device can not only store these sequences in the reinforcement learning data cache but also in the reward model data cache to construct reward experience data. Thus, when the robot performs human-in-the-loop real-machine reinforcement learning training based on the action data sequences in the reinforcement learning data cache, the terminal device can also optimize the trained reward model based on the reward experience data in the reward model data cache. This allows the reward model to continue optimizing during the interaction between the model and the environment.

[0096] In some embodiments, during the online interactive process of the robot performing real-machine reinforcement learning, when the terminal device collects expert interaction action data of the robot, it further generates random actions and their corresponding rewards corresponding to the expert interaction action data. Then, it generates a random action data sequence (s, a, r) ​​based on the random action and its corresponding reward, and only puts this random action data sequence into the reward model data cache for further optimization of the reward model.

[0097] Please refer to Figure 9 , Figure 9 The overall process diagram of the robot real-machine reinforcement learning method provided in this application is shown in some embodiments.

[0098] like Figure 9As shown, in some embodiments, the terminal device's real-machine reinforcement learning training of the robot may include four stages: expert trajectory data acquisition, reward model training, human-in-the-loop real-machine reinforcement learning, and deployment of the reinforcement learning model. Thus, after completing the real-machine reinforcement learning training of the robot through the technical content described in the above embodiments, the terminal device can also deploy the robot's reinforcement learning model.

[0099] It should be noted that, as Figure 10 As shown, during the reinforcement learning model deployment phase, the terminal device no longer needs a reward model and a probabilistic image data augmentation module; instead, it directly uses the original image as model input. Thus, this image background augmentation method leverages the advantages of model background generalization: the data generalization operation only occurs during the training phase, significantly reducing computational load during actual deployment. Furthermore, it is unaffected by detection or segmentation model failures, thereby increasing the model's robustness and resistance to interference.

[0100] The following is a general and complete embodiment of the robot real-machine reinforcement learning method provided in this application.

[0101] In this embodiment, assuming the robot is a six-axis robotic arm, and the robotic arm needs to complete the interactive task of grasping a cube on a table, then action 'a' can be designed as the six-dimensional spatial velocity 'a' of the robot's end effector = ( , , , , , The robotic arm's low-level controller translates the control signals to each joint to achieve end-effector speed control. After the detection or segmentation model identifies a cube in the environment, the background image behind the cube is randomized according to a certain probability p. Then, when a human uses a handle to control the robotic arm to grasp the cube, image data and robotic arm joint speed information s are continuously collected during the grasping process. Human intervention actions are also analyzed. and random actions All rewards are calculated according to the reward rules and awarded accordingly. and rewards Then, the two sequences and All data is stored in the reward model data cache for subsequent training of the reward model. Human intervention action sequences are also stored in the reinforcement learning data cache. Then, supervised learning is performed using data from the reward model data cache. Under this reward rule, expert actions are considered rewardable by default, while rewards for random actions are only used as negative samples for subsequent reward model training. This reward model will output a positive reward (e.g., 0.1) when it sees state s and performs an expert action. Performing an action contrary to the expert action will output a negative reward (e.g., -0.1) to encourage the model to refer to the expert action. Upon completion of the grasping task, the reward model outputs a large positive reward (e.g., 10).

[0102] After completing the reward model training, the robot enters the real-device reinforcement learning phase. The policy model generates the six-dimensional spatial velocity 'a' at the robot's end effector to control the robotic arm's interaction with the environment. Human experts are ready to intervene at any time in case of abnormal situations. Intervention data is stored in both the reinforcement learning data cache and the reward model data cache. Random action data is stored in the reward model data cache, and data generated by the policy model is stored in the reinforcement learning data cache. During the interaction, if it is an expert intervention action or a randomly generated action, it is scored by the reward rules. If it is a policy-generated action, it is scored by the reward model. Through continuous interaction, both the policy model and the reward model are trained.

[0103] In the final deployment phase, reward model calculations and random data augmentation calculations are disabled. The original image and the robot arm joint velocity information are used directly as input to the policy model. The policy model outputs the six-dimensional spatial velocity of the robot's end effector to complete the grasping task. The model trained after background randomization can complete the grasping task on desktops of different colors and textures.

[0104] Please refer to Figure 11 This application also provides a robot real-machine reinforcement learning device, which can implement the above-mentioned robot real-machine reinforcement learning method.

[0105] like Figure 11 As shown, the robot real-machine reinforcement learning device provided in this application may include: The acquisition module is used to acquire environmental visual information when the robot performs interactive tasks in a real physical environment; The background enhancement module is used to perform image background enhancement processing on the environmental visual information to obtain the target environmental visual information with enhanced background. The real-machine reinforcement learning module is used to construct environmental interaction experience data based on the visual information of the target environment, and to perform real-machine reinforcement learning on the robot based on the environmental interaction experience data.

[0106] In some embodiments, the target environment visual information is obtained by image background enhancement processing on the environment visual information in the first environment state observation data, and the first environment state observation data is the environment state observation data of the robot when responding to human control in the real physical environment. The real-machine reinforcement learning module is further used to train the reward model based on expert interaction action data of the robot responding to human control in the real physical environment and visual information of the target environment, to obtain a trained reward model; in response to the robot interacting with the real physical environment, to generate dense reward data of the robot based on the trained reward model; and to construct environmental interaction experience data of the robot based on the dense reward data.

[0107] In some embodiments, the real-device reinforcement learning module is further configured to acquire autonomous interaction action data of the robot; the autonomous interaction action data is the interaction action data generated by the robot's reinforcement learning model when the robot interacts with the real physical environment; based on the trained reward model, the autonomous interaction action data and the second environmental state observation data corresponding to the autonomous interaction action data are processed to obtain the robot's dense reward data; the environmental state observation data includes at least the environmental visual information when the robot interacts with the real physical environment.

[0108] In some embodiments, the real-device reinforcement learning module is further configured to generate an autonomous interaction action data sequence based on the autonomous interaction action data, the second environmental state observation data, and the dense reward data; the environmental interaction experience data includes the interaction action data sequence.

[0109] In some embodiments, the real-machine reinforcement learning module is further used to optimize the trained reward model based on expert interaction action data when the robot responds to human control and interacts with the real physical environment.

[0110] In some embodiments, the real-machine reinforcement learning module is further configured to generate an expert interaction action data sequence based on the robot's expert interaction action data, the third environmental state observation data corresponding to the expert interaction action data, and rule reward information; construct reward experience data based on the expert interaction action data sequence; and optimize the trained reward model based on the reward experience data.

[0111] In some embodiments, the background enhancement module is further configured to randomly extract a portion of the environmental image data to be processed from the environmental visual information; perform background segmentation processing on the environmental image data, and perform image data enhancement processing on the segmented background image data to obtain background-enhanced target environmental image data; the target environmental visual information includes the target environmental image data.

[0112] It should be noted that the specific implementation of the robot real machine reinforcement learning device provided in this application is basically the same as the specific implementation of the robot real machine reinforcement learning method described above, and will not be repeated here.

[0113] Please see Figure 12 This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described robot real-machine reinforcement learning method.

[0114] In some embodiments, the electronic device can be any smart terminal such as a tablet computer, smartphone, in-vehicle hardware platform (e.g., in-vehicle computer), or wearable device.

[0115] like Figure 12 As shown, the electronic device provided in this application may include: The processor 1201 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to achieve the technical solution provided in this application. The memory 1202 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1202 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1202 and is called and executed by the processor 1201 to implement the robot real-machine reinforcement learning method of this application. The input / output interface 1203 is used to implement information input and output; The communication interface 1204 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1205 transmits information between various components of the device (e.g., processor 1201, memory 1202, input / output interface 1203, and communication interface 1204); The processor 1201, memory 1202, input / output interface 1203 and communication interface 1204 are connected to each other within the device via bus 1205.

[0116] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described robot real-machine reinforcement learning method.

[0117] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0118] This application also provides a computer program product, including a computer program, the steps of which are implemented when the computer program is executed by a processor, and are basically the same as the specific embodiments of the robot real machine reinforcement learning method described above, and will not be repeated here.

[0119] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of this application, and do not constitute a limitation on the technical solutions provided in this application. Those skilled in the art will know that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

[0120] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0121] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0122] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0123] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0124] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0125] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0126] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0127] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0128] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0129] The preferred embodiments of this application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of this application shall be within the scope of the claims.

Claims

1. A real-machine reinforcement learning method for robots, characterized in that, The method includes: Acquire environmental visual information when the robot performs interactive tasks in a real physical environment; The environmental visual information is subjected to image background enhancement processing to obtain the target environmental visual information with enhanced background. Environmental interaction experience data is constructed based on the visual information of the target environment, and the robot is subjected to real-machine reinforcement learning based on the environmental interaction experience data.

2. The method according to claim 1, characterized in that, The target environment visual information is obtained by image background enhancement processing of the environmental visual information in the first environmental state observation data. The first environmental state observation data is the environmental state observation data of the robot when responding to human control in the real physical environment. The construction of environmental interaction experience data based on the visual information of the target environment includes: The reward model is trained based on the expert interaction action data of the robot in response to human control in the real physical environment and the visual information of the target environment to obtain the trained reward model. In response to the robot interacting with the real physical environment, dense reward data for the robot is generated based on the trained reward model; The robot's environmental interaction experience data is constructed based on the dense reward data.

3. The method according to claim 2, characterized in that, The generation of dense reward data for the robot based on the trained reward model includes: Acquire the autonomous interaction action data of the robot; the autonomous interaction action data is the interaction action data generated by the robot's reinforcement learning model when the robot interacts with the real physical environment; Based on the trained reward model, the autonomous interaction action data and the corresponding second environmental state observation data are processed to obtain the robot's dense reward data; the environmental state observation data includes at least the environmental visual information when the robot interacts with the real physical environment.

4. The method according to claim 3, characterized in that, The construction of the robot's environmental interaction experience data based on the dense reward data includes: An autonomous interaction action data sequence is generated based on the autonomous interaction action data, the second environmental state observation data, and the dense reward data; the environmental interaction experience data includes the interaction action data sequence.

5. The method according to claim 2, characterized in that, The method further includes: Based on the expert interaction action data of the robot responding to human control and interacting with the real physical environment, the trained reward model is optimized.

6. The method according to claim 5, characterized in that, The optimization of the trained reward model based on expert interaction action data when the robot responds to human control and interacts with the real physical environment includes: Based on the expert interaction action data of the robot, the third environmental state observation data corresponding to the expert interaction action data, and the rule reward information, an expert interaction action data sequence is generated. Reward experience data is constructed based on the expert interaction action data sequence, and the trained reward model is optimized based on the reward experience data.

7. The method according to any one of claims 1 to 6, characterized in that, The image background enhancement processing of the environmental visual information includes: Randomly extract a portion of the environmental image data to be processed from the environmental visual information; The environmental image data is subjected to background segmentation processing, and the segmented background image data is subjected to image data enhancement processing to obtain background-enhanced target environment image data; the target environment visual information includes the target environment image data.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the robot real-machine reinforcement learning method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the robot real-machine reinforcement learning method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the robot real-machine reinforcement learning method as described in any one of claims 1 to 7.

Images