Robot imitation learning system and method based on multi-modal somatosensory data

Abstract

The invention discloses a robot imitation learning system and method based on multi-modal somatosensory data, and belongs to the field of imitation learning. A somatosensory data acquisition device is utilized to capture and analyze the multi-modal data of a human body when the human body executes a specific task in real time, and imitation learning training is carried out based on the data; and obtaining a strategy that the robot can execute the same task. Therefore, the method not only simplifies the data acquisition process and improves the efficiency, but also keeps very high imitation learning precision, so that the robot can more accurately reproduce complex actions of human beings.

Description

Technical Field

[0001] The present invention relates to the technical field of imitation learning, and in particular to a robot imitation learning system and method based on multimodal somatosensory data. Background Art

[0002] In the field of robotics, in order for robots to perform corresponding actions according to instructions, they need to learn human actions. Traditional imitation learning methods usually rely on remote-controlled robot arms or hands to collect operation data, which is not only cumbersome and costly, but also inefficient. In addition, this method usually has difficulty in accurately capturing the details and subtle movements of human operations, resulting in limited learning effects. Summary of the invention

[0003] The present invention provides a robot imitation learning system and method based on multimodal somatosensory data, which not only simplifies the data collection process and improves efficiency, but also maintains a high imitation learning accuracy, so that the robot can more accurately reproduce the complex movements of humans.

[0004] A first aspect of the present invention provides a robot imitation learning system based on multimodal somatosensory data, comprising:

[0005] A somatosensory data collection module, used to collect multimodal somatosensory data;

[0006] A data processing and conversion module is used to reposition the modal somatosensory data so that the movements of each body joint of the robot are consistent with human movements in space, convert the repositioned multimodal somatosensory data into point cloud data, and add the perception state of the robot to the point cloud data for point cloud diffusion, so as to obtain the robot's action strategy and the robot's action instruction sequence;

[0007] The training module is used to train the robot model using the robot's action strategy and the robot's action instruction sequence, so that the trained robot model can imitate human actions.

[0008] Optionally, in one embodiment of the present invention, the somatosensory data acquisition module includes a data acquisition glove and a full-body somatosensory device;

[0009] The data collection gloves are provided with a variety of sensors to collect motion data of various joint positions of the hand;

[0010] The whole body body sensing device includes a variety of sensors and cameras, which are used to collect motion data of whole body movements and postures.

[0011] Optionally, in one embodiment of the present invention, the data processing and conversion module is specifically used to map the full multimodal somatosensory data into the posture of the robot using an inverse kinematics algorithm and to map the fingertip position of a human hand and the palm posture with six degrees of freedom into the hand of the robot.

[0012] Optionally, in one embodiment of the present invention, the data processing and conversion module is specifically used to convert the relocated multimodal somatosensory data into point cloud data, and integrate the virtual point cloud of the robot model into the converted point cloud data through forward kinematics.

[0013] Optionally, in one embodiment of the present invention, the data processing and conversion module is specifically used to generate an action strategy based on the processed point cloud data and in combination with the perception state of the robot, convert the point cloud observation and the robot's state data into a series of preset target positions, and output an instruction sequence as the robot's action.

[0014] A second aspect of the present invention provides a robot imitation learning method based on multimodal somatosensory data, comprising the following steps:

[0015] Collect multimodal somatosensory data;

[0016] Repositioning the modal somatosensory data so that the movements of the robot's body joints are consistent with human movements in space, converting the repositioned multimodal somatosensory data into point cloud data, and adding the robot's perception state to the point cloud data for point cloud diffusion to obtain the robot's action strategy and the robot's action instruction sequence;

[0017] The robot model is trained using the robot's action strategy and the robot's action instruction sequence so that the trained robot model can imitate human actions.

[0018] Optionally, in one embodiment of the present invention, the collecting of multimodal somatosensory data includes collecting motion data of hand joint positions and motion data of whole body movements and postures.

[0019] Optionally, in one embodiment of the present invention, relocating the modality body sensing data includes:

[0020] The inverse kinematics algorithm is used to map the full multimodal somatosensory data into the robot's posture and to map the fingertip positions and six-degree-of-freedom palm posture of the human hand into the robot's hand.

[0021] Optionally, in one embodiment of the present invention, converting the relocated multimodal body sensing data into point cloud data includes:

[0022] The relocated multimodal somatosensory data is converted into point cloud data, and the virtual point cloud of the robot model is integrated into the converted point cloud data through forward kinematics.

[0023] Optionally, in one embodiment of the present invention, the perception state of the robot is added to the point cloud data for point cloud diffusion to obtain the robot's action strategy and the robot's action instruction sequence, including:

[0024] Based on the processed point cloud data and combined with the robot's perception state, an action strategy is generated, which converts the point cloud observation and the robot's state data into a series of preset target positions and outputs a command sequence as the robot's action.

[0025] The robot imitation learning system and method based on multimodal somatosensory data of the embodiments of the present invention perform imitation learning through multimodal somatosensory data, which not only simplifies the data collection process and improves efficiency, but also maintains a high imitation learning accuracy, so that the robot can more accurately reproduce complex human movements.

[0026] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0028] Figure 1 A schematic diagram of the structure of a robot imitation learning system based on multimodal somatosensory data provided according to an embodiment of the present invention;

[0029] Figure 2 A schematic diagram of a robot imitation learning process of multi-modal somatosensory data according to an embodiment of the present invention;

[0030] Figure 3 A schematic diagram of a robot imitation application of multimodal somatosensory data according to an embodiment of the present invention;

[0031] Figure 4 4 is a flowchart of a robot imitation learning method based on multimodal somatosensory data according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0033] Figure 1 A schematic diagram of the structure of a robot imitation learning system based on multimodal somatosensory data provided according to an embodiment of the present invention.

[0034] like Figure 1 As shown, the robot imitation learning system 10 based on multimodal somatosensory data includes: a somatosensory data acquisition module 100, a data processing and conversion module 200 and a training module 300.

[0035] Among them, the somatosensory data acquisition module 100 is used to collect multimodal somatosensory data. The data processing and conversion module 200 is used to reposition the modal somatosensory data so that the movements of each body joint of the robot are consistent with human movements in space, convert the repositioned multimodal somatosensory data into point cloud data, and add the robot's perception state to the point cloud data for point cloud diffusion to obtain the robot's action strategy and the robot's action instruction sequence. The training module 300 is used to train the robot model using the robot's action strategy and the robot's action instruction sequence so that the trained robot model imitates human movements.

[0036] In one embodiment of the present invention, the somatosensory data acquisition module includes a data acquisition glove and a whole-body somatosensory device; the data acquisition glove is provided with a variety of sensors to collect motion data of each joint position of the hand; the whole-body somatosensory device includes a variety of sensors and cameras to collect motion data of whole-body movements and postures.

[0037] Specifically, the data collection gloves are equipped with multiple highly sensitive sensors that can accurately record the movements of each joint of the hand, including fingertip pressure and the detailed angle of finger bending. The whole body sensing device integrates multiple sensors and cameras to fully capture the wearer's whole body movements and postures, and can record in detail various movements from head to toe, such as gait, body tilt angle, and relative position between limbs, thereby providing all-round data support for imitation learning.

[0038] In one embodiment of the present invention, the data processing and conversion module is specifically used to map the full multimodal somatosensory data into the robot's posture using an inverse kinematics algorithm and to map the fingertip position of a human hand and the palm posture of a hand with six degrees of freedom into the robot's hand.

[0039] The key to relocalizing hand and body motions is to adjust and convert the collected human hand and whole body motion data to adapt to the robot's operational requirements. Due to the structural differences between humans and robots, it is challenging to directly transfer these motion data to the robot. In particular, hand motions require accurate mapping of the fingertip positions of the human hand and the palm posture of the six degrees of freedom (6-DoF) to the robot's hand. In addition, the whole body motion data is also processed by the inverse kinematics (IK) algorithm to ensure that the robot's body joint movements are consistent with human movements in space. This includes adjusting the optimal position and direction of the robot's wrist, elbow, shoulder and other joints to ensure the continuity and coordination of the movement. These redirected motion data will be used as labels for robot training to ensure that the robot can accurately imitate the details of human movements.

[0040] In one embodiment of the present invention, the data processing and conversion module is specifically used to convert the relocated multimodal somatosensory data into point cloud data, and integrate the virtual point cloud of the robot model into the converted point cloud data through forward kinematics.

[0041] In observation data post-processing, RGB-D images captured by the LiDAR camera are converted into point cloud data. This conversion improves the stability of the observation data and enhances the flexibility and operability of the data. To ensure that the point cloud data is consistent with the visual representation of human hand and body movements, the virtual point cloud of the robot model is integrated into the real observation point cloud through forward kinematics. This point cloud data is then used to train the robot's motion policy to ensure that the captured actions are executable within the robot's operating range. In addition, all RGB-D frames are processed into point clouds aligned with the robot space, and task-irrelevant elements such as desktop points are removed, thereby optimizing the input observation data for the robot policy.

[0042] In one embodiment of the present invention, the data processing and conversion module is specifically used to generate an action strategy based on the processed point cloud data and in combination with the perception state of the robot, convert the point cloud observation and the robot's state data into a series of preset target positions, and output an instruction sequence as the robot's action.

[0043] In an embodiment of the present invention, the point cloud diffusion strategy is a key part, which aims to train the robot's motion strategy so that it can accurately imitate and perform complex human movements. The strategy uses the processed and converted point cloud data, combined with the robot's current self-perception state (including the position information of the hands, wrists and other body joints), to generate motion strategies. The point cloud diffusion strategy converts point cloud observations and the robot's state data into a series of preset target positions and outputs a sequence of instructions as the robot's actions. This strategy not only generates a coherent motion trajectory, but also optimizes the smoothness and naturalness of the motion, thereby greatly improving training efficiency and execution accuracy.

[0044] The main advantage of the point cloud diffusion strategy is that it can effectively handle high-dimensional multimodal motion instructions. This strategy is particularly suitable for complex tasks in the field of robot learning, such as fine manipulation and multi-joint coordination, because it can: Comprehensively process high-dimensional data: By integrating high-dimensional point cloud data with the robot's multimodal perception state, the strategy can generate more complex and detailed action sequences; Adapt to complex action spaces: By adjusting point clouds and action trajectories through data augmentation techniques (such as random 2D translation), the point cloud diffusion strategy enhances the robot's adaptability and generalization to complex action spaces; Improve learning flexibility and accuracy: The diffusion strategy model allows flexible adjustment of parameters and states during training, thereby accurately imitating human actions and improving the accuracy of task execution.

[0045] like Figure 2 and Figure 3 As shown, the robot imitation learning system based on multimodal somatosensory data of the present invention includes three parts:

[0046] 1. Data collection:

[0047] Hand and body motion capture, where operators wear data gloves and full-body sensory devices to perform specific tasks. These devices capture the operator's hand movements and full-body posture data, recording every subtle movement and corresponding spatial position;

[0048] 2. Data processing and conversion

[0049] Hand and body motion repositioning, using inverse kinematics (IK) technology to process hand and body motion data to adapt to the robot's motion execution system;

[0050] Observation data post-processing, by converting RGB-D images into point clouds, stabilizes the observation data and optimizes its performance within the robot's operating space. The point cloud data is further refined to exclude any elements not relevant to the task, providing clear and focused input data;

[0051] 3. Strategy training and execution:

[0052] Point cloud diffusion strategy training: Using the converted and optimized point cloud data, combined with the robot's self-perception state, the robot's action execution strategy is trained through the point cloud diffusion strategy.

[0053] In an embodiment of the present invention, in order to map the human hand and body posture data to the robot's joint configuration, inverse kinematics (IK) is used to calculate the robot's joint angles:

[0054] \[\textbf{q}_t=\text{IK}(\textbf{p}_h)\]

[0055] Here, \(\textbf{p}_h\) is the position and posture data of the human end effector, and \(\textbf{q}_t\) represents the joint angle that the robot should achieve at time \(t\).

[0056] Point cloud conversion:

[0057] The data collected through RGB-D images is converted into point clouds \(\textbf{O}_t\), and then data enhancement processing is performed as the robot's visual perception:

[0058] \[\textbf{O}_t=\text{PC}(\text{RGB-D}_t)\]

[0059] Here, the \(\text{PC}\) function processes the RGB-D image \(\text{RGB-D}_t\) collected by the camera and converts it into point cloud data.

[0060] Point cloud diffusion strategy training: Based on the point cloud data \(\textbf{O}_t\) and the robot's current joint state \(\textbf{q}_t\), the strategy model \(\pi\) generates the following action sequence \(\textbf{a}_t\):

[0061] \[\textbf{a}_t=\pi(\textbf{q}_t,\textbf{O}_t)\]

[0062] \(\textbf{q}_t\) represents the current joint state of the robot, \(\textbf{O}_t\) is the processed point cloud data, and \(\textbf{a}_t\) is the action instruction output by the policy model \(\pi\) based on the current state and observation.

[0063] Position control and motion output:

[0064] Based on the action sequence \(\textbf{a}_t\) generated by the policy model \(\pi\), the robot performs the corresponding actions:

[0065] \[\textbf{q}_{t+1}=\text{Execute}(\textbf{a}_t)\]

[0066] Here, the function \(\text{Execute}\) updates the robot's joint states to the next time step \(\textbf{q}_{t+1}\) according to the provided action instructions \(\textbf{a}_t\).

[0067] The robot imitation learning system based on multimodal somatosensory data of the present invention is described below through specific embodiments.

[0068] 1. System Configuration

[0069] Hardware equipment: inspection robot, somatosensory data acquisition module and high-resolution camera (used to capture the operator's hand movements when pressing buttons and detailed location data).

[0070] Imitation learning module: A deep learning model deployed on a robot or cloud server to learn and replicate human fine hand operations.

[0071] Executive control system: A control unit integrated on the robot that is responsible for executing actions based on the learned model output.

[0072] 2. Inspection scenario description

[0073] The robot is deployed in a power control station equipped with multiple operating buttons to perform routine operation tasks. The task goal is to imitate the actions of human technicians and accurately press the designated buttons to execute specific power system control commands.

[0074] 3. Operation process

[0075] Mission start: When the operation mission begins, the technician demonstrates the action of pressing buttons through the data gloves, and the robot captures the detailed data of these actions in real time through its observation module.

[0076] Data collection: The somatosensory data collection module captures the technician's hand movements and button position data, recording every detail of the movement, such as the bending angle of the fingers and the pressing force.

[0077] Imitation learning training: The imitation learning module analyzes the collected hand motion data and trains the robot to imitate these movements through a point cloud diffusion strategy.

[0078] Action Execution: The execution control system controls the robot arm and hand to perform the same button pressing action based on the output of the imitation learning module. The system dynamically adjusts the robot's hand position and strength to ensure the accuracy of the press.

[0079] Performance feedback and optimization: If an execution error is observed, such as an unsuccessful button press, the robot will automatically record the parameters of this operation and adjust its action strategy to optimize subsequent execution results.

[0080] 4. Model Application

[0081] In this power inspection application scenario, the key to imitation learning is to replicate human hand movements with high precision. By accurately capturing and learning how technicians press buttons, the robot can autonomously perform complex control tasks, reduce direct human intervention, and improve the safety and efficiency of operations. In addition, by continuously collecting operation data and feedback, the system can continuously optimize the action execution strategy and improve the accuracy and adaptability of the robot's operations.

[0082] This specific implementation demonstrates how to apply advanced imitation learning technology to actual industrial scenarios, effectively solving delicate operation problems that are difficult for traditional automation systems to handle, and significantly improving the operational capabilities and intelligence level of automation systems.

[0083] The robot imitation learning system based on multimodal somatosensory data proposed in an embodiment of the present invention performs imitation learning through multimodal somatosensory data, which not only simplifies the data collection process and improves efficiency, but also maintains a high imitation learning accuracy, so that the robot can more accurately reproduce complex human movements.

[0084] Figure 4 4 is a flowchart of a robot imitation learning method based on multimodal somatosensory data according to an embodiment of the present invention.

[0085] like Figure 4 As shown, the robot imitation learning method based on multimodal somatosensory data includes the following steps:

[0086] In step S101, multimodal body sensing data is collected.

[0087] In step S102, the modal somatosensory data is repositioned so that the movements of the robot's body joints are consistent with human movements in space, the repositioned multimodal somatosensory data is converted into point cloud data, and the robot's perception state is added to the point cloud data for point cloud diffusion to obtain the robot's action strategy and the robot's action instruction sequence.

[0088] In step S103, the robot model is trained using the robot's motion strategy and the robot's motion instruction sequence, so that the trained robot model imitates human motion.

[0089] Optionally, in one embodiment of the present invention, the collecting of multimodal somatosensory data includes collecting motion data of hand joint positions and motion data of whole body movements and postures.

[0090] Optionally, in one embodiment of the present invention, relocating the modality body sensing data includes:

[0091] The inverse kinematics algorithm is used to map the full multimodal somatosensory data into the robot's posture and to map the fingertip positions and six-degree-of-freedom palm posture of the human hand into the robot's hand.

[0092] Optionally, in one embodiment of the present invention, converting the relocated multimodal body sensing data into point cloud data includes:

[0093] The relocated multimodal somatosensory data is converted into point cloud data, and the virtual point cloud of the robot model is integrated into the converted point cloud data through forward kinematics.

[0094] Optionally, in one embodiment of the present invention, the perception state of the robot is added to the point cloud data for point cloud diffusion to obtain the robot's action strategy and the robot's action instruction sequence, including:

[0095] Based on the processed point cloud data and combined with the robot's perception state, an action strategy is generated, which converts the point cloud observation and the robot's state data into a series of preset target positions and outputs a command sequence as the robot's action.

[0096] According to the robot imitation learning method based on multimodal somatosensory data proposed in an embodiment of the present invention, imitation learning is performed through multimodal somatosensory data, which not only simplifies the data collection process and improves efficiency, but also maintains a high imitation learning accuracy, so that the robot can more accurately reproduce the complex movements of humans.

[0097] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0098] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0099] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

Claims

1. A robot imitation learning system based on multimodal somatosensory data, characterized in that: include: A somatosensory data collection module, used to collect multimodal somatosensory data; A data processing and conversion module is used to reposition the modal somatosensory data so that the movements of each body joint of the robot are consistent with human movements in space, convert the repositioned multimodal somatosensory data into point cloud data, and add the perception state of the robot to the point cloud data for point cloud diffusion, so as to obtain the robot's action strategy and the robot's action instruction sequence; The training module is used to train the robot model using the robot's action strategy and the robot's action instruction sequence, so that the trained robot model can imitate human actions.

2. The system according to claim 1, characterized in that The somatosensory data acquisition module includes data acquisition gloves and full-body somatosensory equipment; The data collection gloves are provided with a variety of sensors to collect motion data of various joint positions of the hand; The whole body body sensing device includes a variety of sensors and cameras, which are used to collect motion data of whole body movements and postures.

3. The system according to claim 1, characterized in that The data processing and conversion module is specifically used to map the full multimodal somatosensory data into the robot's posture using an inverse kinematics algorithm and to map the fingertip position of a human hand and the palm posture with six degrees of freedom into the robot's hand.

4. The system according to claim 1, characterized in that The data processing and conversion module is specifically used to convert the relocated multimodal somatosensory data into point cloud data, and integrate the virtual point cloud of the robot model into the converted point cloud data through forward kinematics.

5. The system according to claim 4, characterized in that The data processing and conversion module is specifically used to generate an action strategy based on the processed point cloud data and in combination with the perception state of the robot, convert the point cloud observation and the robot's state data into a series of preset target positions, and output an instruction sequence as the robot's action.

6. A robot imitation learning method based on multimodal somatosensory data, characterized in that: The following steps are involved: Collect multimodal somatosensory data; Repositioning the modal somatosensory data so that the movements of the robot's body joints are consistent with human movements in space, converting the repositioned multimodal somatosensory data into point cloud data, and adding the robot's perception state to the point cloud data for point cloud diffusion to obtain the robot's action strategy and the robot's action instruction sequence; The robot model is trained using the robot's action strategy and the robot's action instruction sequence so that the trained robot model can imitate human actions.

7. The method according to claim 6, characterized in that The collecting of multimodal somatosensory data includes collecting motion data of the positions of various joints of the hand and motion data of whole body movements and postures.

8. The method according to claim 6, characterized in that Relocating the modal body sensing data includes: The inverse kinematics algorithm is used to map the full multimodal somatosensory data into the robot's posture and to map the fingertip positions and six-degree-of-freedom palm posture of the human hand into the robot's hand.

9. The method according to claim 6, characterized in that Convert the relocated multimodal somatosensory data into point cloud data, including: The relocated multimodal somatosensory data is converted into point cloud data, and the virtual point cloud of the robot model is integrated into the converted point cloud data through forward kinematics.

10. The method according to claim 9, characterized in that The robot's perception state is added to the point cloud data for point cloud diffusion to obtain the robot's action strategy and the robot's action instruction sequence, including: Based on the processed point cloud data and combined with the robot's perception state, an action strategy is generated, which converts the point cloud observation and the robot's state data into a series of preset target positions and outputs a command sequence as the robot's action.

Images