A robot wiping action policy learning method based on vision and force perception

Abstract

The application discloses a robot wiping action strategy learning method based on vision and force perception, a robot wiping action planning model is constructed by using a PPO algorithm, the PPO limits the amplitude of policy update by using a method of calculating a certain truncation ratio, ensures that the new policy does not deviate too far from the old policy, and thus finds a balance between exploration and utilization. This design helps to maintain the stability of the learning process and avoid large performance decline. Through the training in the dense simulation environment and the continuous optimization of the reward mechanism, the mechanical arm will gradually refine its wiping skills, not only improving the task execution efficiency, but also flexibly coping with different cleaning needs under the premise of ensuring safety. Finally, the mature trained model is deployed to the entity mechanical arm for field testing and fine-tuning, which significantly enhances the wiping operation ability of the mechanical arm in diversified home application scenarios.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of home service robot action planning method, and particularly relates to a robot wiping action strategy learning method based on vision and force perception. BACKGROUND

[0002] Mechanical arms, as a kind of machine devices with multiple degrees of freedom and programmable, have been widely used in the field of industrial automation, performing diversified tasks such as assembly, welding and transportation. With the development of technology, the application of mechanical arms has expanded from traditional industrial production to warehouse logistics, medical care and home service fields. In particular, in the home environment, the application of mechanical arms has brought new challenges and requirements, such as the need to operate in a more complex and dynamic environment while ensuring high efficiency and safety.

[0003] The mechanical arm applied in home service is different from that applied in industry, and the robot needs to perform tasks in a non-structured environment. If the traditional control method is used, it means an increase in the state space and action space dimensions for the robot, an exponential growth in the number of control parameters, and an increasing difficulty in motion control.

[0004] The application mainly faces the field of home service and studies a robot action adaptive planning method based on multi-modal perception information. At present, there are problems in applying mechanical arms to the home service industry: (1) Limited adaptability to environmental changes. The home environment belongs to an unstructured environment or an environment with large disturbances. According to the different environments of each family, the use of traditional methods to plan the action of the mechanical arm needs to be adjusted for each family, and the adjustment effect may not be ideal. (2) Complex design. If the mechanical arm wants to achieve smooth action through traditional algorithms, it often needs to design a complex structure and adjust dozens of parameters, and it is not easy to make targeted adjustments when problems occur.

[0005] Compared with the above methods, reinforcement learning can accumulate data during the training process, train itself with its own data, and automatically adjust the strategy to adapt to dynamic and complex environments through interaction and feedback with the environment. Deep reinforcement learning can process high-dimensional data through neural networks, making it feasible to handle complex tasks in complex environments. When transplanted to a new environment to complete the same task, reinforcement learning continuously improves performance through continuous learning and updating of strategies, adapting to new tasks and environments. SUMMARY

[0006] To solve the above problems, the application aims to provide a robot wiping action strategy learning method based on vision and force perception, which introduces an advanced algorithm in the field of deep reinforcement learning, PPO (Proximal Policy Optimization). As a policy gradient method, PPO effectively balances the relationship between exploration and utilization by using the concept of trust region in the policy update process, thereby greatly improving the learning efficiency while maintaining the stability of learning. The action planning model constructed by the PPO algorithm can more finely optimize the wiping action strategy of the robot in a complex environment, ensuring that the robot arm does not degrade in performance due to overly aggressive policy updates while quickly adapting to environmental changes and continuously optimizing its behavior.

[0007] The technical solution adopted by the application is a robot wiping action strategy learning method based on vision and force perception. The PPO algorithm is used to construct a robot wiping action planning model. The reason for choosing PPO algorithm for improvement is that PPO algorithm has better stability compared to other reinforcement learning algorithms. PPO limits the magnitude of policy updates by using a method of calculating a certain truncation ratio, ensuring that the new policy does not deviate too far from the old policy, thereby finding a good balance between exploration and utilization. This design helps maintain the stability of the learning process and avoid significant performance degradation. The clipping strategy of PPO algorithm is as follows:

[0008]

[0009] Where t represents the time step in the wiping process, theta is the parameter of the PPO algorithm strategy, which determines the behavior of the next step in the wiping process according to the trained strategy, r t (theta) is the ratio of new and old strategies; is the advantage function estimate, which measures the relative advantage of taking action a compared to the average case, and epsilon is the clipping coefficient used to limit the magnitude of policy changes. The Clip operation compares r t (theta) with the size of the limit range to control the policy change within a certain range, ensuring that the policy space can be effectively explored during training while maintaining the stability and efficiency of training.

[0010] The reward function part designed in the application will be directly applied to the calculation of the advantage function A(s, a). The calculation of the advantage function is as follows:

[0011] A(s, a) = Q(s, a) - V(s)

[0012] Wherein, Q(s, a) represents the sum of the expected cumulative rewards after taking the wiping action a in the current state s of the robot arm. The reward function designed this time will be mainly applied in the calculation of Q(s, a). V(s) is the value of the state s, and the value of V(s) is generated by the value network after judging the state.

[0013] The specific measures are: first, a simulation environment containing rich details is constructed to accurately simulate the robot arm operation live, including but not limited to integrating robot arm hardware state monitoring, visual data processing and real-time feedback of force sensing information. Then, the collected multi-modal data is preprocessed, such as image gray scale conversion, size adjustment and cropping, and force sensing signal normalization, to facilitate efficient processing by the model. On this basis, the PPO model is configured and optimized, which not only involves careful design of the model architecture - using convolutional neural networks to process image input, and multilayer perceptron to process vector data such as force sensing, but also focuses on adjusting learning rate, network layers and other hyperparameters, and designing a refined reward function that can reflect the quality of task completion, execution efficiency, and safety, to guide the PPO algorithm to efficiently learn the wiping strategy.

[0014] Compared with the prior art, the method of the present application is trained in a dense simulation environment and continuously optimized by the reward mechanism. The robot arm will gradually refine its wiping skills, not only improving task execution efficiency, but also flexibly responding to different cleaning needs while ensuring safety. Finally, the trained model is deployed to the physical robot arm for field testing and fine-tuning. Although there will be a gap between simulation and reality, thanks to the robustness of the PPO algorithm and the previous fine-tuning, the robot arm can quickly adapt to changes in the real environment, significantly enhancing its wiping operation capabilities in diverse home application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is a flowchart of how the simulation part of the present application calculates the erasing area and the six-dimensional moment of force;

[0016] Figure 2 is a program flowchart of how the simulation part of the present application publishes whiteboard pictures and updates the whiteboard;

[0017] Figure 3 is a logic flowchart of collision detection and robot arm reset in the simulation environment of the present application;

[0018] Figure 4 is a reinforcement learning reward calculation flowchart of the present application;

[0019] Figure 5 is a design diagram of how to set up time stamp synchronization of multi-sensor data in the present application;

[0020] Figure 6is the feature extractor workflow picture in the application;

[0021] Figure 7 is the training result picture in the simulation environment of the application;

[0022] Figures 8-11 is the wiping effect demonstration in the simulation environment of the application.

[0023] Figure 8 is the mechanical arm approaching the whiteboard and starting to wipe.

[0024] Figure 9 is the mechanical arm erasing the third and fourth stains.

[0025] Figure 10 is the calculation of whether all erasures are complete.

[0026] Figure 11 is the mechanical arm returning to the initial position after wiping. DETAILED DESCRIPTION

[0027] The application will be described in detail below in combination with the drawings and examples.

[0028] The technical solution adopted by the application proposes the application of the proximal policy optimization algorithm in deep reinforcement learning to train the mechanical arm to perform the whiteboard wiping task in a complex environment. In the design, multi-modal data, including vision and touch, are input to help the mechanical arm better perceive the surrounding environment. The visual input is used to identify the cleanliness and environmental characteristics, while the touch data assist the mechanical arm to perceive the force and stability of its wiping action. By adjusting the reward mechanism of reinforcement learning to optimize the model, the mechanical arm has the ability of self-adaptation and continuous improvement in the process of completing the task, so as to achieve better adaptability.

[0029] The application faces the wiping task in the home scenario, and the task planning and operation generation will be realized through the following steps:

[0030] (1) Simulate the real environment in the simulation environment. In order to ensure that the simulation environment is highly close to the actual situation, so as to smoothly realize the conversion from simulation to entity environment, and apply the design to the real scene, the corresponding code needs to be developed to reproduce the interactive situation in the real environment. This simulation experiment adopts two sets of programs, which aims to simulate the actual conditions in detail and perform related calculations. Figure 1 Focus on calculating the eraser cleaning area, force and torque, and broadcast the polygon vertices and mechanical parameters formed by erasing through the topic mechanism of ROS for use by other system components and PPO algorithm.

[0031] The program architecture is based on a base class named `ObjectBase`, which is responsible for defining the spatial basic information of the object in the erasing training (such as the center position, direction, size, and rotation matrix), and providing methods for calculating the vertices of the cube and constructing the plane equation, laying the foundation for subsequent operations. Through the DH parameter method, the precise positioning of the cube corner points in the base coordinate system is realized. On this basis, the `WhiteBoard` and `Erase` classes are derived, corresponding to the specific operations of the whiteboard and eraser, respectively. Both of them receive size parameters through ROS topics when instantiated.

[0032] The calculation process of the erasing area contains multiple steps of geometric analysis: first, the intersection of the two surfaces is solved using the plane equation; then, by projecting the eraser and analyzing the intersection points of the intersection line and the whiteboard projection plane, a series of ordered vertices describing the erasing polygon are formed. To solve the problem of false positives caused by the back approach of the eraser, distance judgment logic is added to ensure that the shadow area is only calculated in reasonable situations. In addition, due to the nonlinearity of the ROS standard torque sensor output, the program takes a different approach by simulating the erasing force through the volume intrusion estimation method, and combining the force arm to calculate the torque value, which is published through the topic to simulate the output of the six-dimensional force sensor, serving the PPO training.

[0033] Attached Figure 2 Focusing on the ROS publishing and dynamic updating of the stain image, based on the polygon vertex coordinates provided by the previous program, the erasing effect is presented on the image in real time, and the PPO reset instruction is responded to, restoring the image to the original state without stains. The program executes the image publishing task through an independent thread, with a frequency of once per second, ensuring that the data is transmitted to the ` / board / image_array` topic, and maintaining the consistency and safety of image data access through the thread lock mechanism. When receiving the polygon vertex data, the callback function is activated to complete the drawing and filling of the polygon on the image, while the reset signal triggers another callback function to achieve the instant recovery of the image without restarting the program, ensuring the continuity and efficiency of the simulation process.

[0034] (2) Design the experimental environment. In order to train a general model for the wiping task, some settings need to be made for reinforcement learning. The core of reinforcement learning involves two environmental elements: `step` and `reset`. `step` receives actions, and feeds back observations of the next state, rewards, termination signals, and additional information, among which the reward function is the key, which guides algorithm learning by quantifying goals, acts as a bridge between goals and algorithms, and affects learning efficiency and effectiveness. Setting rewards requires breaking down the task into sub-goals and providing appropriate rewards and punishments. `reset` is triggered by the `done` signal, indicating that the termination condition has been reached, such as a collision of the robotic arm or completion of the task, and the environment will be reset. To complete this task, the following conditions are specified for the reset condition: First, collisions of the robotic arm and the base of the eraser are defined as unwanted situations, and the working range of the end is limited. In reality, the range of exploration of the robotic arm can also be limited. Add the three situations to the termination condition in the step function to form four termination conditions: 1. All pixel points in the published whiteboard stain picture are background color (white), i.e. it has been wiped. 2. The base of the eraser collides. 3. The robotic arm collides. 4. The end of the robotic arm exceeds the specified range of motion. To avoid conflicts on the model due to collisions of the robotic arm, a multi-threaded approach is used to accept information about collisions of the robotic arm and the base of the eraser. Two threads run simultaneously outside the main thread. If a collision is detected, the robotic arm position is immediately reset. The reset process is shown in FIG. 8. Figure 3

[0035] In the `step` element of the environment, the reward function reward is specially set. When the robotic arm or the eraser collides, it may be that the robotic arm has a large motion amplitude and the eraser is too deep, which may represent that the eraser has collided with the whiteboard. If only the position is reset, it will lead to sparse rewards, which may again cause the occurrence of local optimal conditions. It is a reasonable choice to make further logical judgments.

[0036] ​Therefore, the reward logic designed for the wiping task is as follows: first, the system checks whether the robot arm exceeds its predetermined range of activity. If the robot arm exceeds the range of activity, a reward of -50 will be returned. If the robot arm does not exceed the range of activity, the process will continue. At this time, the system will check the flags flagl and flag2 as collision flags and eraser flags. According to the state of these flags, the system will give different reward values. When the robot arm collision is detected, it is judged whether the eraser has contacted the whiteboard before the collision. If it has contacted the whiteboard, a positive reward of 5x the contact area is returned; if it has not contacted, a penalty is given, reward = -50. If no robot arm collision is detected from the flags, but the eraser collision is detected, a positive reward of 10x the contact area is returned, and if there is no contact, the reward is assigned to -10. If neither of the two collisions occurs, the whiteboard state is judged. If the eraser leaves a wiping mark, a reward of 50x the number of wiping pixels is returned. If only the eraser contacts the whiteboard, a reward of 30x the wiping area is given. If none of the above conditions occurs, the robot arm is in an exploration state, and a reward of 0.1 is given.

[0037] In the final stage of the process, regardless of whether the robot arm exceeds the range of activity or meets which conditions, the system will eventually return to the reward variable, i.e. the reward value in the entire operation process. The reward value is used to evaluate the action given by PPO. The overall setting of the step part is shown in the accompanying Figure 4

[0038] At the same time, in order to ensure the synchronization of data transmission in the wiping training process, multi-sensor data timestamp synchronization is added in the training environment. The topic type is changed in the program of publishing images and shadow area, and the time of publishing the topic is added on the basis of the original content. The time of receiving the reset topic and publishing the reset picture is added in the training environment, and the time of publishing the reset topic is recorded. If the time of the whiteboard data is less than the reset time, wait for the whiteboard to deliver new data, and execute the reset of the environment after the data synchronization. At the same time, time stamp and synchronization waiting are also added to all communication contents to avoid the occurrence of asynchronization. The logic of timestamp synchronization is shown in the accompanying Figure 5

[0039] ​​(3) Design feature extractor. In deep reinforcement learning, it is necessary to integrate the environment state and extract features through a neural network. In the wiping design, image, torque sensor and mechanical arm joint position input are involved. Different structures are needed to process different input types. For image type input, a CNN with three convolutional layers is used for feature extraction, with filter numbers of 32, 64 and 64 respectively, convolution kernel numbers of 8x8, 4x4 and 3x3 respectively, and strides of 4, 2 and 1 respectively. For pure digital type vector input, such as torque and wiping area information, a MLP with two layers is used for extraction. Finally, a MLP with two fully connected layers is used for integration, and the integrated features are input to the PPO algorithm. The design of the feature extractor is shown in Figure 5 .

[0040] In addition, the parameters of PPO also need to be designed. The clipping rate is 0.2, the number of interactions per update cycle is 1024, the number of iterations is 10, the batch size is 128, and the total number of interactions is 102400 steps.

[0041] Finally, training is performed. The mechanical arm motion planning model that can complete the wiping task in the simulation environment is obtained. The curve in the training process is shown in Figure 7 . The wiping effect is shown in the attached Figures 8-11 .

[0042] (4) Apply the trained model to the actual mechanical arm and real environment. The model trained in the simulation environment has certain generalization ability after multiple iterations. Next, the model needs to be imported into the actual mechanical arm and applied to the real environment for further training. Considering the possible delay and bandwidth limitation, develop an interface layer to ensure that the model trained in the simulation environment can seamlessly communicate with the actual hardware. In addition, before deploying the model, calibrate the mechanical arm: calibrate the camera's intrinsic and extrinsic parameters. Calibrate the torque sensor's zero point to ensure that the sensor outputs zero when there is no external force; and calibrate the sensitivity by applying a known size and direction of torque to measure the sensor's response, to determine the sensor's sensitivity and establish a correction curve to compensate for any nonlinear effects. Replace the simulation data obtained with the timestamp synchronization design with the data obtained by the sensors in the real environment, to ensure that the timestamps of the RGB-D camera and the torque sensor are aligned, and that the data obtained from different sensors can be fused in a unified coordinate system. Through the above steps, the accuracy of the sensor data and the precision of the motion execution are ensured. The model that achieves the effect in the simulation environment is migrated to the actual environment for further training, and a mechanical arm motion planning model with certain generalization ability in the real environment is obtained.

Claims

1. A method for learning a robot wiping action policy based on vision and force perception, the method comprising: A robot wiping action planning model is constructed using the PPO algorithm, and the clipping strategy of the PPO algorithm is as follows: ； Wherein t represents the time step in the wiping process; θ is the parameter of the PPO algorithm strategy, which determines the behavior of the next step in the wiping process according to the trained strategy; Is the ratio of new and old strategies; Is the advantage function estimate value, which measures the relative advantage of taking action a compared to the average case; ε is the clipping coefficient, which is used to limit the policy change range; the Clip operation compares With the size of the limit range to control the policy change within a certain range, ensuring that the policy space can be effectively explored during training, and the stability and efficiency of training can be maintained. The reward function part is applied directly to the computation of the advantage function The computation of the advantage function is: ； wherein, represents the sum of the expected cumulative rewards after taking the wiping action a in the current state s of the mechanical arm; the reward function designed this time will be mainly applied in the calculation of ; is the value of the state s, The value of is generated by the value network after judging the state; first, a simulation environment is constructed to simulate the actual operation of the mechanical arm, including integrated mechanical arm hardware state monitoring, visual data processing, and real-time feedback of force sensing information; then, the collected multi-modal data is preprocessed, including image grayscale conversion, size adjustment and cropping, and force sensing signal normalization, to facilitate efficient processing by the model; The robot faces the wiping task in the home scene, and the task planning and motion generation are realized through the following steps: Step (1), simulate the real environment in the simulation environment; develop the corresponding code to reproduce the interactive situation in the real environment; the simulation experiment adopts two sets of programs, simulates the actual conditions and performs related calculations; calculate the erasing area, force and torque of the eraser, and broadcast the polygon vertex and mechanical parameters formed by erasing through the topic mechanism of ROS; the calculation process of the erasing area includes multi-step geometric analysis: first, solve the intersection of two surfaces using the plane equation; then, by projecting the eraser and analyzing the intersection of the intersection line and the whiteboard projection plane, a series of ordered vertices describing the erasing polygon are formed; to solve the misjudgment caused by the back approach of the eraser, distance judgment logic is added to ensure that the shadow area is calculated only in the approach situation; the erasing force is simulated by the volume intrusion estimation method, and the torque value is calculated by combining the force arm, which is published through the topic to simulate the output of the six-dimensional force sensor, serving the PPO training; Step (2), design the experimental environment; set the reinforcement learning to train a general model for the wiping task; Step (3), design the feature extractor; integrate the environment state and extract the features through the neural network; Step (4), apply the trained model to the actual mechanical arm and real environment; import the model trained in the simulation environment into the actual mechanical arm, and apply the real environment for further training; migrate the model achieving the effect in the simulation environment to the actual environment, and train again to obtain a mechanical arm action planning model with generalization ability in the real environment.

2. The robot wiping action strategy learning method based on vision and force perception according to claim 1, characterized in that, In step (1), the program architecture of the simulation experiment is based on a base class named `ObjectBase`, which is responsible for defining the spatial basic information of the objects in the wiping training and providing methods for calculating the cube vertex and constructing the plane equation; the cube corner is positioned in the base coordinate system through the DH parameter method; the `WhiteBoard` and `Erase` classes are derived, corresponding to the specific operations of the whiteboard and eraser, and receive size parameters through the ROS topic when instantiated; Focus on the ROS publishing and dynamic updating of the stain image, and according to the coordinates of the polygon vertex, the erasing effect is presented on the image in real time, and the PPO reset instruction is responded to restore the image to the original state without stains; The image publishing task is executed through an independent thread, and the frequency is set to one per second to ensure that the data is transmitted to the ` / board / image_array` topic, and the thread lock mechanism is used to maintain the consistency and safety of image data access; when receiving the polygon vertex data, the callback function is activated to complete the drawing and filling of the polygon on the image, and the reset signal triggers another callback function to realize the instant recovery of the image; In step (2), the reinforcement learning core involves two environment elements: `step` and `reset`. `step` receives actions, feeds back observations of the next state, rewards, termination signals, and additional information. The reward function guides algorithm learning by quantifying goals. Setting rewards requires breaking down tasks into sub-goals and providing appropriate rewards and punishments. `reset` is triggered by the `done` signal, indicating that the termination condition has been met. In the `step` environment element, the reward function `reward` is set as follows: First, the system checks whether the robot arm has exceeded its predetermined activity range. If the robot arm has exceeded the activity range, a reward of -50 is returned. If the robot arm has not exceeded the activity range, the process continues. The flags `flag1` and `flag2`, which serve as collision and eraser flags, are checked. Different reward values are given based on the states of these flags. When a collision is detected, the system determines whether the eraser has contacted the whiteboard before the collision. If the eraser has contacted the whiteboard, a positive reward of 5 times the contact area is returned. If there has been no contact, a penalty of -50 is given. If a collision is not detected from the flags but an eraser collision is detected, a positive reward of 10 times the contact area is returned. If there has been no contact, a reward of -10 is assigned. If neither type of collision has occurred, the whiteboard state is determined. If an eraser mark is left, a reward of 50 times the number of erased pixels is returned. If only the eraser has contacted the whiteboard, a reward of 30 times the erased area is given. If none of these conditions have occurred, the robot arm is in an exploratory state, and a reward of 0.1 is given. At the end of the process, regardless of whether the robot arm has exceeded the activity range or met any of the conditions, the system will eventually return to the reward variable, which is the reward value throughout the operation process. The reward value is used to evaluate the action provided by PPO. To ensure the synchronization of data transmission during the wiping training process, multiple sensor data timestamps are synchronized in the training environment. The topic type is changed in the program that publishes images and shadow areas, and the time of publishing the topic is added. The time of receiving the reset topic and publishing the reset image is added in the training environment, and the time of publishing the reset topic is recorded. If the time of the whiteboard data is less than the reset time, the system waits for the whiteboard to transmit new data and performs the environment reset after the data is synchronized. Time stamps and synchronization waiting are added to all communication content. In step (3), in the wiping design, inputs of images, torque sensors and mechanical arm joint positions are involved; for image type input, a CNN with three convolutional layers is used for feature extraction, the number of filters in each layer is 32, 64 and 64 respectively, the number of convolution kernels is 8x8, 4x4 and 3x3 respectively, and the stride is 4, 2 and 1 respectively; for the pure digital type of vector input, a MLP with two layers is used for extraction; finally, a MLP with two fully connected layers is used for integration, and the integrated features are input to the PPO algorithm; the parameters of the PPO algorithm are designed; the clipping rate is 0.2, the number of interaction steps in each update cycle is 1024, the number of iterations is 10, the batch size is 128, and the total number of interaction steps is 102400 steps; finally, the training is performed to obtain the mechanical arm motion planning model for completing the wiping task in the simulation environment; In step (4), the mechanical arm is calibrated, the simulation data obtained by the multi-sensor data timestamp synchronization design is replaced by the data obtained by the sensor in the real environment, the timestamps of the RGB-D camera and the torque sensor are aligned, and the data obtained from different sensors are fused in a unified coordinate system; the accuracy of the sensor data and the accuracy of the motion execution are ensured.

3. The robot wiping motion policy learning method based on vision and force perception according to claim 2, characterized in that, In the designed experimental environment, three situations are added to the step function termination condition to form four termination conditions: 1) all pixel points in the published whiteboard stain picture are background color, that is, it has been wiped; 2) the eraser root collides; 3) the mechanical arm collides; 4) the end of the mechanical arm exceeds the specified motion range; the multi-thread mode is used to accept the mechanical arm collision and eraser root collision information; Two threads are running simultaneously outside the main thread; if a collision is detected, the mechanical arm position is immediately reset. ​

Images