Multi-modal fusion and control platform and system for interaction of human and humanoid robot
Through multi-sensor fusion and coordinate calibration technology, combined with generalized embodied intelligent algorithms, the problem of insufficient motion recognition and prediction accuracy in interaction between humans and humanoid robots is solved, and more stable and efficient human-computer interaction is achieved.
Patent Information
- Application Number
- CN202410957225.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-11-08
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the existing human-human-like robot interaction technology, the accuracy of action recognition and prediction is insufficient, the data acquisition stability and reliability of multiple sensors are poor, and the sensor coordinate calibration is complex, making it difficult to achieve efficient action prediction.
Using multi-sensor fusion module, spatial coordinate calibration fusion module, adaptive filtering module and generalized embodied intelligent human-computer interaction algorithm model, combining human signal fusion perception platform and double-arm robot mirror control platform, through multi-sensor data fusion and coordinate calibration and filtering technology to improve the accuracy and adaptability of action recognition and prediction.
It significantly improves the stability and reliability of data acquisition, reduces the complexity of platform logic, enhances the ability to recognize and predict actions, and makes human-computer interaction more natural and accurate.
Smart Images

Figure CN118906046A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of robot interaction technology, and in particular to a multi-modal fusion and control platform and system for human-humanoid robot interaction. Background Art
[0002] In the existing human-computer interaction technology, the interaction between humans and humanoid robots faces many technical challenges. In order to achieve accurate, comprehensive and continuous motion recognition and prediction in complex human-robot interaction tasks, it is necessary to deploy a multi-sensor system for more multimodal and comprehensive perception. Multi-sensors can capture information from different dimensions in different forms, such as video image signals, inertial signals, muscle electrical signals, etc. These information complement each other and can provide more comprehensive interaction data.
[0003] However, it is difficult to collect stable and continuous data on human interaction. The limitations of the sensors themselves and the interference of the external environment will lead to insufficient stability and reliability of data collection. In addition, each of the multi-sensor sensors related to posture has its own independent reference coordinate system, which requires spatial coordinate calibration and unification. This process is complex and time-consuming because the position and orientation of each sensor need to be accurately calibrated to ensure the accuracy of the data. This increases the logical complexity of the platform, especially when multiple sensors are involved.
[0004] The layout of multiple sensors on the human body and robots needs to be carefully screened and designed. A reasonable layout can not only improve the accuracy of data collection, but also effectively reduce interference and redundant information between sensors. However, in practical applications, how to arrange these sensors to obtain more accurate motion prediction functions is still a difficult problem.
[0005] The accuracy of current motion recognition and prediction algorithms needs to be improved. Although some advanced algorithms can achieve high recognition rates under ideal conditions, in practical applications, motion recognition and prediction in complex interactive scenarios still face huge challenges. Complex interactive scenarios not only involve the recognition and prediction of multiple actions, but also factors such as environmental changes and the diversity of interactive objects, which puts higher demands on the adaptability and generalization ability of the algorithm.
[0006] To this end, the present invention aims to provide a human-humanoid robot interaction platform to solve the above problems. Summary of the invention
[0007] The purpose of the present invention is to solve the above problems and to provide a multimodal fusion and control platform and system for the interaction between humans and humanoid robots, including a human signal fusion perception platform and a dual-arm robot mirror control platform. Through multi-sensor arrangement and fusion, a stable multi-sensor coordinate calibration system, adaptive filtering technology and a generalized embodied intelligent human-computer interaction algorithm model, the interaction quality between humans and humanoid robots in motion recognition and prediction is improved.
[0008] In order to achieve the above object, the technical solution of the present invention is as follows:
[0009] The present invention provides a multi-modal fusion and control platform and system for human-humanoid robot interaction, the system comprising a multi-sensor fusion module, a space coordinate calibration fusion module, an adaptive filtering module and a human-computer interaction algorithm module;
[0010] The multi-sensor fusion module is used to collect and synchronize data from multiple sensors;
[0011] The spatial coordinate calibration and fusion module is used to calibrate and unify the coordinate systems of different sensor posture data;
[0012] The adaptive filtering module is used to optimize the processing of sensor data, reduce noise and improve data accuracy;
[0013] The human-machine interaction algorithm module contains a generalized embodied intelligence algorithm for analyzing human actions and predicting corresponding robot responses. The system is highly flexible and adaptable, and can be configured and adjusted according to different application task requirements, thereby improving operational efficiency and task completion quality.
[0014] The platform includes a human signal fusion perception platform and a dual-arm robot mirror control platform profile frame;
[0015] The human body signal fusion sensing platform includes a human body signal fusion sensing platform profile frame, motion sensing wearable gloves and a surface electromyography multi-lead sensing bracelet;
[0016] The human signal fusion perception platform profile frame and the dual-arm robot mirror control platform profile frame are both rectangular frames. The human signal fusion perception platform profile frame is divided into two parts, the upper part is the depth camera visual acquisition space, and the operator places the upper limbs in the depth camera visual acquisition space in the human signal fusion perception platform profile frame, and wears the device to collect relevant signals and electromyographic information;
[0017] The human signal fusion perception platform is equipped with multiple redundant sensing devices, such as several non-contact depth cameras and contact motion sensing wearable gloves and surface electromyography multi-lead sensing bracelets, to fuse the multi-modal data of the upper limbs, improve the response speed and accuracy of the humanoid robot to human movements, and thus enhance the naturalness and fluency of human-computer interaction.
[0018] The multiple depth cameras collect the operator's hand movement information in a multi-angle non-contact manner, which can avoid occlusion problems. The multiple depth cameras can be freely moved and placed on the human signal fusion perception platform profile frame, and the positions of the depth cameras can be adjusted based on needs.
[0019] The motion sensing gloves are equipped with an integrated inertial navigation unit and a bending sensor. When the operator wears the motion sensing gloves, the operator's hand motion information is collected in a contact manner to enhance the robustness of the system's refined control; the surface electromyography multi-lead sensing bracelet is worn on the forearms and upper arms of the operator's arms to sense the motion state information of the upper limbs and arms of the human body, and to obtain multi-lead electromyography signals of the upper limbs and arms of the human body.
[0020] A data center is provided at the bottom of the profile frame of the human signal fusion perception platform, and the data center includes a display and several depth camera industrial computers;
[0021] The dual-arm robot mirror control platform profile frame is provided with a dual-arm robot mirror control platform controller, and a depth camera is provided above the dual-arm robot mirror control platform controller for sensing the interactive environment and information of the interactive object. The dual-arm robot mirror control platform controller is provided with humanoid mechanical arms on opposite sides, and the end of the humanoid mechanical arm not connected to the dual-arm robot mirror control platform controller is provided with a humanoid robot dexterous hand for performing designated tasks. The humanoid mechanical arm imitates the movement of human arms; the humanoid robot dexterous hand is used for fine operation and grasping; the dual-arm robot mirror control platform controller is used to receive and process control signals;
[0022] The humanoid mechanical arm and the humanoid robot dexterous hand of the dual-arm robot mirror control platform profile frame can both imitate the movement of human upper limbs under control, wherein the fingers of the humanoid robot dexterous hand have tactile feedback information.
[0023] The bottom of the human signal fusion perception platform profile frame and the dual-arm robot mirror control platform profile frame are provided with universal wheels for the two frames to move and freely arrange.
[0024] The dual-arm robot mirror control platform is mainly used to receive the operator's hand, forearm and upper arm movement information and multimodal perception data obtained by the human signal fusion perception platform, so as to realize the refined control or interactive feedback of the remote control operation of the robotic arms on both sides of the dual-arm robot mirror control platform. Through the fusion processing of these signals and data, the platform can accurately imitate the operator's actions, or realize interaction and collaboration with people, and is used to complete designated tasks remotely. The platform is suitable for various complex environments that require remote operation, including but not limited to: medical surgery assistance; operations in dangerous environments, such as nuclear power plants, chemical plants, etc.; external maintenance of spacecraft, etc.
[0025] The control logic flow steps of the present invention include:
[0026] S1. The multimodal fusion and control platform for human-humanoid robot interaction is launched.
[0027] S2. The operator stands in the workstation, wearing motion sensing gloves and surface electromyography multi-lead sensor bracelet. After the platform and sensor are initialized, they begin to collect the upper limb movement status and muscle information of the operator in the workstation in real time.
[0028] S3, pre-processing, adaptive filtering, world coordinate system alignment, and multi-modal data fusion of data collected in real time by multiple sensors and the deep visual data and dexterous hand tactile data fed back by the dual-arm robot mirror control platform. The processed data obtained include the operator's hand movement information, upper limb muscle status, upper limb posture information, and the visual information and tactile force information of the dual-arm robot mirror control platform.
[0029] S4. Input the data into the human-computer interaction algorithm module, enter the personalized human-computer interaction scene and start the specific human-computer interaction task.
[0030] S5, the depth camera of the dual-arm robot mirror control platform obtains the depth information of the current visual range in real time and feeds it back to S3 for data processing.
[0031] S6. The humanoid robotic arm and dexterous hand of the dual-arm robot mirror control platform respond based on the output of the human-computer interaction algorithm.
[0032] S7: The tactile feedback signal of the dexterous hand is fed back to S3 for data processing.
[0033] S8. Realize embodied intelligent human-computer interaction applications.
[0034] Compared with the prior art, this solution has the following beneficial effects:
[0035] The present invention significantly improves the stability and reliability of data acquisition through multi-sensor fusion and arrangement. The stable multi-sensor coordinate calibration system reduces the logical complexity of the platform and improves the accuracy of sensor data. Adaptive filtering technology is used to effectively filter noise and provide more accurate motion data. The generalized embodied intelligent human-computer interaction algorithm model enhances the robot's action recognition, prediction and foresight capabilities in interaction, enabling it to interact with humans more naturally and accurately. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 Schematic diagram of the structure of the human-humanoid robot interaction platform in an embodiment of the present invention;
[0037] Figure 2 It is the overall data flow and analysis flow chart of the platform in the embodiment of the present invention;
[0038] Figure 3 It is a specific control logic block diagram of the platform in an embodiment of the present invention.
[0039] In the figure: 1. Human signal fusion perception platform profile frame; 2. Dual-arm robot mirror control platform profile frame; 3. Surface electromyography multi-lead sensing bracelet; 4. Motion sensing wearable gloves; 5. Depth camera; 6. Display; 7. Depth camera industrial computer; 8. Universal wheel; 9. Humanoid robot dexterous hand; 10. Dual-arm robot mirror control platform controller; 11. Humanoid robotic arm. DETAILED DESCRIPTION
[0040] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solution of the present invention will be further described in detail below in conjunction with the embodiments of the present invention and the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0041] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will be described in detail below in conjunction with the embodiments.
[0042] Example:
[0043] The human signal fusion perception platform and the dual-arm robot mirror control platform frames are both rectangular frames. The size of the dual-arm robot mirror control platform frame is larger than the robot's spatial range to ensure the safety of movement in the space. A data center is provided at the bottom of the human signal fusion perception platform frame. The data center includes a main control display 6 and several depth camera 5 industrial computers. Different numbers of depth camera 5 industrial computers are configured and used according to the number of sensors in the frame. At the same time, the display 6 can be used for host debugging and data visualization.
[0044] The dual-arm robot mirror control platform controller 10 mainly receives the operator's hand, forearm and upper arm motion information obtained by the human body signal fusion perception platform, as well as multimodal perception data, so as to realize the remote control operation of the mechanical arms on both sides of the dual-arm robot mirror control platform controller 10. Through the fusion processing of these signals and data, the dual-arm robot mirror control platform controller 10 can accurately imitate the operator's actions, or realize interaction and collaboration with people, and is used to complete designated tasks remotely. The dual-arm robot mirror control platform controller 10 is suitable for various complex environments that require remote operation, including but not limited to medical surgery assistance, operations in dangerous environments (such as nuclear power plants, chemical plants, etc.) and external maintenance of spacecraft.
[0045] The system's multi-sensor fusion module is used to collect and integrate data from multiple sensors; the spatial coordinate calibration fusion module is used to correct and synchronize the coordinate systems of different sensor data; the adaptive filtering module is used to optimize the processing of sensor data, reduce noise and improve data accuracy; the human-computer interaction algorithm module contains a generalized embodied intelligence algorithm to analyze human actions and predict the corresponding robot response. Use multiple sensors to collect effective data within the local range of motion (humans include: deep visual information, surface electromyography information, finger curvature data and inertial navigation unit information; robots include: deep visual information, force sensor information). Then perform different signal processing on different information, such as segmentation, filtering, feature extraction, and multi-modal information interaction after aligning the spatial coordinates of multiple depth cameras. Select and build different human-computer interaction intention perception models according to actual application requirements (such as trajectory planning, human-computer interaction, human-computer collaboration and human-computer integration).
[0046] In order to ensure that sensors with direction perception functions can perform accurate spatial coordinate calibration, the present invention has developed a stable calibration program. The calibration program can perform high-precision coordinate calibration and change of multiple depth cameras through the spatial posture information and spatial confidence provided, realize the world coordinate system, and improve the overall performance of the invention.
[0047] Several depth cameras 5 are arranged on the top of the human signal fusion perception platform profile frame 1 to realize multi-angle non-contact collection of operator's hand motion information, which can avoid occlusion problems. Several depth cameras 5 are driven by motors on the human signal fusion perception platform profile frame 1, and the depth camera 5 collection position can be adjusted based on application requirements. The operator wears motion sensing gloves 4, which are used to collect the operator's hand motion information in contact and enhance the robustness of the system's refined control. The surface electromyography multi-lead sensing bracelet 3 is worn on the forearms and upper arms of the operator's arms on both sides to sense the motion state information of the upper limbs and arms of the human body, and at the same time obtain the electromyography signals of the upper limbs and arms of the human body, which are used to monitor the operator's fatigue level and prevent fatigue operation.
[0048] A dual-arm robot mirror control platform controller 10 is provided at the top center of the dual-arm robot mirror control platform profile frame 2, which completes the integration, connection and processing of all components and is also the center for human interaction; a depth camera 5 is provided at the top of the dual-arm robot mirror control platform profile frame 2, and humanoid mechanical arms 11 are provided on both sides of the dual-arm robot mirror control platform controller 10, and the end of the humanoid mechanical arm 11 is connected to a humanoid robot dexterous hand 9 for performing designated tasks. The humanoid robot dexterous hand 9 is used for fine operation and grasping; the humanoid mechanical arm 11 imitates the movement of human arms; the dual-arm robot mirror control platform controller 10 is used to receive and process control signals; the depth camera 5 is used to perceive the interactive environment and the information of the interactive object. The human signal fusion perception platform profile frame 1 is equipped with multiple redundant sensing devices, such as non-contact several depth cameras 5 and contact motion perception wear gloves 4 to fuse the perception of hand data, so as to improve the response speed and accuracy of the humanoid robot to human actions, thereby enhancing the naturalness and fluency of human-computer interaction.
[0049] The system is highly flexible and adaptable, and can be configured and adjusted according to different task requirements, thereby improving operational efficiency and task completion quality. The motion sensing wearable glove 4 is equipped with an integrated inertial navigation unit and a bending sensor. The operator wears the motion sensing wearable glove 4 and the surface electromyography multi-lead sensing bracelet 3, and places the upper limb in the human upper limb signal sensing frame space, which can collect arm motion state information, finger bending and electromyography information.
[0050] In addition, a wheeled intelligent sensing chassis is provided at the bottom of the human signal fusion perception platform profile frame 1 and the dual-arm robot mirror control platform profile frame 2, including inertial sensors and torque sensors, which are used to monitor and adjust the movement status of the platform in real time to ensure the stability and flexibility of the platform in different terrains and environments.
[0051] Through the above design, the present invention provides an efficient, flexible and precise multi-modal fusion and control platform and system for human-humanoid robot interaction, which is suitable for various application scenarios that require refined operations and adaptation to complex environments.
[0052] When the device is used through the above-mentioned embodiments of the present invention, it can be configured in a variety of ways according to the needs of the operator. If the operator needs to achieve high-precision and complete long-term hand interaction, after entering the motion capture space and wearing the equipment, a series of hand movements are completed in the space. In addition, if the motion action involves the use of other instruments, such as holding other tools, different sensing devices can complement each other. The depth camera and the motion sensing gloves can complement each other in motion pattern recognition. For example, the acceleration and angular velocity data provided by the motion sensing gloves can supplement the shortcomings of the camera system in occlusion situations. At the same time, in a specific movement, the platform can focus on the body parts involved in the action. For example, when performing gestures, you can focus on the position and motion data of the palm to reduce unnecessary data redundancy.
[0053] The robot responds based on the perceived operator data according to the specific human-machine interaction task requirements, including the response of the mechanical arms and the response of the five-fingered dexterous hands. In addition, in the robot platform, the data information perceived by the depth camera and the tactile information of the five-fingered dexterous hands are fed back to the embodied intelligent human-machine interaction algorithm model to realize the closed loop of the process and achieve a true interactive effect.
[0054] The above specific embodiments are merely explanations of the present invention and are not limitations of the present invention. After reading this specification, those skilled in the art may make modifications to the embodiments without any creative contribution as needed. However, such modifications are protected by the patent law as long as they are within the scope of the claims of the present invention.
Claims
1. A multimodal fusion and control platform for human-humanoid robot interaction, characterized by: A profile frame (2) comprising a human signal fusion perception platform and a dual-arm robot mirror control platform; The human body signal fusion sensing platform comprises a human body signal fusion sensing platform profile frame (1), a surface electromyography multi-lead sensing bracelet (3), and a motion sensing wearable glove (4); The human body signal fusion perception platform profile frame (1) and the dual-arm robot mirror control platform profile frame (2) are both rectangular frames. A data center is provided at the bottom of the human body signal fusion perception platform profile frame (1), and the data center includes a display (6) and a plurality of depth camera industrial computers (7); A dual-arm robot mirror control platform controller (10) is provided on the top of the dual-arm robot mirror control platform profile frame (2), and humanoid mechanical arms (11) are provided on opposite sides of the dual-arm robot mirror control platform controller (10), and a humanoid mechanical dexterous hand (9) is provided at one end of the humanoid mechanical arm (11) that is not connected to the dual-arm robot mirror control platform controller (10).
2. A multi-modal fusion and control system for human-humanoid robot interaction as claimed in claim 1, characterized in that: The system includes a multi-sensor fusion module, a spatial coordinate calibration fusion module, an adaptive filtering module and a human-computer interaction algorithm module; The multi-sensor fusion module is used to collect and integrate data from multiple sensors; The spatial coordinate calibration and fusion module is used to calibrate and synchronize the coordinate systems of different sensor data; The adaptive filtering module is used to optimize the processing of sensor data, reduce noise and improve data accuracy; The human-computer interaction algorithm module includes a generalized embodied intelligence algorithm for analyzing human actions and predicting corresponding robot responses.
3. The multimodal fusion and control platform for human-humanoid robot interaction as claimed in claim 1, characterized in that: The human body signal fusion perception platform profile frame (1) and the dual-arm robot mirror control platform controller (10) are both provided with a plurality of cameras on the top, and the cameras are depth cameras (5).
4. The multimodal fusion and control platform for human-humanoid robot interaction as claimed in claim 1, characterized in that: The motion sensing wearable glove (4) is provided with an integrated inertial navigation unit and a bending sensor.
5. The multi-modal fusion and control platform for human-humanoid robot interaction as claimed in claim 4, characterized in that: The operator wears the motion sensing gloves (4) and the surface electromyography multi-lead sensing bracelet (3), and places the upper limbs in the space of the human body signal fusion sensing platform profile frame (1) to collect relevant signals and electromyography information.
6. The multi-modal fusion and control platform for human-humanoid robot interaction as claimed in claim 3, characterized in that: A plurality of the depth cameras (5) are movably connected to the human body signal fusion perception platform profile frame (1).
7. The multi-modal fusion and control platform for human-humanoid robot interaction as claimed in claim 1, characterized in that: Universal wheels (8) are provided at the bottom of the human body signal fusion perception platform profile frame (1) and the dual-arm robot mirror control platform profile frame (2).
Citation Information
Patent Citations
Teleoperation system and method based on exoskeleton data glove and myoelectricity wristband
CN110039507A
Humanoid remote control method for immersion type dual-arm robot
CN111319026A
Man-machine gesture cooperative control method based on myoelectric sensing and visual sensing
CN113947815A
Robot humanoid impedance control method and system based on multi-modal signal fusion
CN116300436A
Exoskeleton device and system for teleoperation of humanoid robot and teaching method
CN117921638A
