Multi-modal fusion and control platform and system for interaction of human and humanoid robot

Abstract

The invention relates to the technical field of robot interaction, and discloses a multi-modal fusion and control platform and system for human and humanoid robot interaction, which comprises a human body signal fusion sensing platform and a double-arm robot mirror image control platform, and has the technical key points that the multi-modal fusion platform system for human and humanoid robot interaction is provided; through arrangement and fusion of multiple sensors, in combination with a stable multi-sensor coordinate calibration system, an adaptive filtering technology and a generalized intelligent man-machine interaction algorithm model, the interaction quality of a person and a humanoid robot in the aspects of action recognition, prediction and tracking is improved; the platform can effectively capture and analyze human actions, and through accurate classification and regression algorithms, the response speed and accuracy of the humanoid robot to the human actions are improved, so that the naturalness and fluency of human-computer interaction are enhanced; in addition, the invention also pays attention to the stability and reliability of the system, and ensures that the interaction quality and experience can be provided in various complex environments.

Description

Technical Field

[0001] The present invention relates to the field of robot interaction technology, and in particular to a multi-modal fusion and control platform and system for human-humanoid robot interaction. Background Art

[0002] In the existing human-computer interaction technology, the interaction between humans and humanoid robots faces many technical challenges. In order to achieve accurate, comprehensive and continuous motion recognition and prediction in complex human-robot interaction tasks, it is necessary to deploy a multi-sensor system for more multimodal and comprehensive perception. Multi-sensors can capture information from different dimensions in different forms, such as video image signals, inertial signals, muscle electrical signals, etc. These information complement each other and can provide more comprehensive interaction data.

[0003] However, it is difficult to collect stable and continuous data on human interaction. The limitations of the sensors themselves and the interference of the external environment will lead to insufficient stability and reliability of data collection. In addition, each of the multi-sensor sensors related to posture has its own independent reference coordinate system, which requires spatial coordinate calibration and unification. This process is complex and time-consuming because the position and orientation of each sensor need to be accurately calibrated to ensure the accuracy of the data. This increases the logical complexity of the platform, especially when multiple sensors are involved.

[0004] The layout of multiple sensors on the human body and robots needs to be carefully screened and designed. A reasonable layout can not only improve the accuracy of data collection, but also effectively reduce interference and redundant information between sensors. However, in practical applications, how to arrange these sensors to obtain more accurate motion prediction functions is still a difficult problem.

[0005] The accuracy of current motion recognition and prediction algorithms needs to be improved. Although some advanced algorithms can achieve high recognition rates under ideal conditions, in practical applications, motion recognition and prediction in complex interactive scenarios still face huge challenges. Complex interactive scenarios not only involve the recognition and prediction of multiple actions, but also factors such as environmental changes and the diversity of interactive objects, which puts higher demands on the adaptability and generalization ability of the algorithm.

[0006] To this end, the present invention aims to provide a human-humanoid robot interaction platform to solve the above problems. Summary of the invention

[0007] The purpose of the present invention is to solve the above problems and to provide a multimodal fusion and control platform and system for the interaction between humans and humanoid robots, including a human signal fusion perception platform and a dual-arm robot mirror control platform. Through multi-sensor arrangement and fusion, a stable multi-sensor coordinate calibration system, adaptive filtering technology and a generalized embodied intelligent human-computer interaction algorithm model, the interaction quality between humans and humanoid robots in motion recognition and prediction is improved.

[0008] In order to achieve the above object, the technical solution of the present invention is as follows:

[0009] In a first aspect, the present invention provides a multimodal fusion and control platform for human-humanoid robot interaction, the multimodal fusion and control platform comprising a human signal fusion perception platform and a dual-arm robot mirror control platform profile frame;

[0010] The human body signal fusion sensing platform includes a human body signal fusion sensing platform profile frame, a surface electromyography multi-lead sensing bracelet, and motion sensing wearable gloves;

[0011] The human signal fusion perception platform profile frame and the dual-arm robot mirror control platform profile frame are both rectangular frames. A data center is provided at the bottom of the human signal fusion perception platform profile frame. The data center includes a display and several depth camera industrial computers.

[0012] A dual-arm robot mirror control platform controller is provided on the top of the dual-arm robot mirror control platform profile frame, and humanoid mechanical arms are provided on opposite sides of the dual-arm robot mirror control platform controller. A humanoid mechanical arm is provided at one end of the humanoid mechanical arm that is not connected to the dual-arm robot mirror control platform controller. The humanoid mechanical arm includes three shoulder joints, one elbow joint and three wrist joints. Each joint of the humanoid mechanical arm adopts a high-precision servo motor and a multi-stage reducer. The humanoid mechanical arm adopts a series configuration and operates based on the form of multi-joint connecting rod follow-up.

[0013] The humanoid robotic dexterous hand comprises six degrees of freedom, wherein the thumb structure comprises two degrees of freedom, including bending degrees of freedom and joint degrees of freedom, which are used for precise grasping and manipulation of complex objects, and the remaining four fingers each comprise one degree of freedom of bending. The humanoid robotic dexterous hand is used for performing finger bending and grasping actions based on a multi-joint linkage follow-up form by integrating a high-precision micro-servo motor and a linkage mechanism, and the humanoid robotic dexterous hand has a built-in force tactile sensor.

[0014] In some embodiments, the human signal fusion perception platform profile frame and the top of the dual-arm robot mirror control platform controller are both equipped with multiple depth cameras.

[0015] In some embodiments, the motion sensing wearable glove includes an inertial sensor and a bending sensor for capturing hand movements;

[0016] There are eleven inertial sensors, which are respectively distributed at the palm position, the metacarpophalangeal joint of each finger, and the proximal phalangeal joint of each finger;

[0017] There are fourteen bending sensors, which are respectively distributed at the metacarpophalangeal joint of each finger, the proximal knuckle of each finger, and the distal knuckles of the four fingers other than the thumb. The bending sensors are used to sense the bending angle of the finger toward the palm and capture the second bending movement of the finger.

[0018] In some embodiments, the depth camera is connected to the human signal fusion perception platform profile frame and the dual-arm robot mirror control platform controller through a slidable device.

[0019] In some embodiments, the bottom of the human signal fusion perception platform profile frame and the dual-arm robot mirror control platform profile frame are both provided with universal wheels.

[0020] In a second aspect, the present invention provides a multimodal fusion and control system for human-humanoid robot interaction, the multimodal fusion and control system comprising a multi-sensor fusion module, a spatial coordinate calibration fusion module, an adaptive filtering module and a human-computer interaction module;

[0021] The multi-sensor fusion module is used to collect and fuse data from multiple sensors to obtain multi-sensor fusion data, and send the multi-sensor fusion data to the spatial coordinate calibration fusion module, wherein the data from the multiple sensors include surface electromyography data collected by a surface electromyography multi-lead sensing bracelet, three-axis acceleration data and three-axis posture data collected by an inertial sensor of a motion sensing glove, and curvature data collected by a curvature sensor, and the multi-sensor fusion data includes fused hand muscle state data and hand motion posture data;

[0022] The spatial coordinate calibration and fusion module is used to calibrate and synchronize the human behavior data in different coordinate systems collected by the depth camera of the human signal fusion platform profile frame, and to perform coordinate correction on the depth visual information collected by the depth camera of the dual-arm robot mirror control platform controller to obtain multi-sensor fusion calibration data, which has the same spatial coordinate system;

[0023] The adaptive filtering module is used to perform adaptive filtering on the human behavior data after correction and synchronization processing in the multi-sensor fusion calibration data and the pressure data collected by the built-in force tactile sensor of the humanoid robot dexterous hand (9), so as to obtain filtered human behavior data and pressure data;

[0024] The human-computer interaction module is used to determine the robot response strategy based on the multi-sensor fusion data, filtered human behavior data and pressure data, and based on the robot response strategy, control the humanoid robotic arm and the humanoid robot dexterous hand to perform interactive movements with the user, wherein the robot response strategy includes the position and posture of the humanoid robotic arm and the humanoid robot dexterous hand.

[0025] In some embodiments, in the multi-sensor fusion module,

[0026] The user's hand motion posture is determined according to the following posture estimation formula:

[0027]

[0028] In the formula, q k-1 and q k are the quaternion-based attitude data at the k-1th moment and the kth moment, represents the quaternion multiplication symbol, Δt is the time interval between the k-1th moment and the kth moment, ω k is the angular velocity data at the kth moment;

[0029] According to the following formula, the acceleration data and attitude data are transformed to obtain the linear acceleration in the global coordinate system:

[0030] α global =R(q k )·α

[0031] Among them, α global is the linear acceleration in the global coordinate system, R(q k ) is q k The generated rotation matrix, α is the acceleration data;

[0032] According to the following formula, the position trajectory of the hand movement is determined:

[0033] v k =v k-1 +α global Δt

[0034] p k =p k-1 +v k Δt

[0035] Where, ν k is the speed of hand movement at the kth moment, p k is the position of the hand movement at the kth moment, v k-1 is the speed of hand movement at the k-1th moment, p k-1 is the position of the hand movement at the k-1th moment;

[0036] According to the following formula, the multi-sensor fusion data is determined:

[0037] θ i =f b (s i )

[0038] P k_fusion =f1(v k ,p k ,θ i )

[0039] Where P k_fusion is the hand motion posture data in the multi-sensor fusion data at the kth moment, f1() is the fusion transformation function, θ i is the curvature data of the ith finger joint at the kth moment, s i is the resistance change of the i-th bending sensor at the k-th moment, f b () is the change mapping function;

[0040] The surface electromyography data collected by the surface electromyography multi-lead sensor bracelet (3) is expressed as E = {E R ,E L}, E={E R ,E L} contains the left hand surface electromyography data and the right hand surface electromyography data, where Represents the data of the i-th lead in the left / right hand surface electromyography data. According to the following formula, the hand muscle state data in the multi-sensor fusion data is determined:

[0041]

[0042] Where M L and M R Represents the hand muscle state data of the left hand and the right hand respectively, f N is the mapping function from surface electromyography data to hand muscle state data.

[0043] In some embodiments, in the spatial coordinate calibration fusion module,

[0044] According to the following formula, the corrected and synchronized human behavior data and the coordinate-corrected depth visual information in the multi-sensor fusion calibration data are determined:

[0045]

[0046] Where W is the defined world coordinate system, C i is the coordinate system of the depth camera itself on the i-th human signal fusion platform profile frame, Indicates that from Ci The transformation matrix from the coordinate system to the W coordinate system, Represents human behavior data in the W coordinate system, Indicates that in C i Human behavior data in coordinate system, P W It represents the human behavior data after correction and synchronization processing in the multi-sensor fusion calibration data in the W coordinate system, f2() is the spatial coordinate fusion function, and n represents the sensor number.

[0047] In some embodiments, in the adaptive filtering module,

[0048] P = f3(P W ,P k_fusion )

[0049] F i =f4(M i )

[0050] Where P is the filtered human behavior data, P k_fusion is the hand motion posture data in the multi-sensor fusion data at the kth moment, P W represents the human behavior data after correction and synchronization processing in the multi-sensor fusion calibration data in the W coordinate system, f3() is the fusion function, f4() is the mapping function used to map the pressure data matrix before filtering to the pressure data after filtering, M i is the pressure data matrix M before filtering at the i-th finger of the humanoid robot dexterous hand i , F i Represents the pressure data of the i-th humanoid robot dexterous hand obtained by filtering.

[0051] In some embodiments, in the human-computer interaction module, the robot response strategy is determined by generalizing the large model embodied intelligence core algorithm; and the angles of each joint of the humanoid robotic arm and the humanoid robot dexterous hand are determined by the robot inverse kinematics solution method.

[0052] Compared with the prior art, this solution has the following beneficial effects:

[0053] The present invention significantly improves the stability and reliability of data collection through multi-sensor fusion and arrangement. Stable multi-sensor coordinate calibration reduces the logical complexity of the platform and improves the accuracy of sensor data. Adaptive filtering technology is used to effectively filter noise and provide more accurate motion data. The generalized embodied intelligent human-computer interaction algorithm model enhances the robot's ability to recognize, predict and foresee actions in interaction, enabling it to interact with humans more naturally and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1Schematic diagram of the structure of a multi-modal fusion and control platform for interaction between humans and humanoid robots in an embodiment of the present invention;

[0055] Figure 2 It is an overall data flow and analysis flow chart of a multi-modal fusion and control platform for interaction between humans and humanoid robots in an embodiment of the present invention;

[0056] Figure 3 It is a control logic flow chart of a multi-modal fusion and control system for interaction between a human and a humanoid robot in an embodiment of the present invention.

[0057] In the figure: 1. Human signal fusion perception platform profile frame; 2. Dual-arm robot mirror control platform profile frame; 3. Surface electromyography multi-lead sensing bracelet; 4. Motion sensing wearable gloves; 5. Depth camera; 6. Display; 7. Depth camera industrial computer; 8. Universal wheel; 9. Humanoid robot dexterous hand; 10. Dual-arm robot mirror control platform controller; 11. Humanoid robotic arm. DETAILED DESCRIPTION

[0058] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solution of the present invention will be further described in detail below in conjunction with the embodiments of the present invention and the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0059] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will be described in detail below in conjunction with the embodiments.

[0060] In a first aspect, the present invention provides a multi-modal fusion and control platform for human-humanoid robot interaction, such as Figure 1 As shown, it includes a human signal fusion perception platform and a dual-arm robot mirror control platform profile frame 2.

[0061] The human signal fusion perception platform is a platform for users to exercise, and the dual-arm robot mirror control platform profile frame 2 is a platform where the humanoid mechanical arm 11 and the humanoid robot dexterous hand 9 that interact with the user are located.

[0062] The human signal fusion sensing platform includes a human signal fusion sensing platform profile frame 1, a surface electromyography multi-lead sensing bracelet 3, and a motion sensing wearable glove 4. The human signal fusion sensing platform profile frame 1 can provide a movement space for the user and provide an installation and sliding position for the sensor. The surface electromyography multi-lead sensing bracelet 3 can collect the user's muscle information, and the motion sensing wearable glove 4 can capture the user's hand movement posture. The user wears the motion sensing wearable glove 4 for contact collection of the user's hand movement posture information to enhance the robustness of the system's refined control. The surface electromyography multi-lead sensing bracelet 3 is worn on the forearms and upper arms of the user's arms on both sides to sense the movement state information of the human upper limbs and arms, and at the same time obtain the electromyography signals of the human upper limbs and arms, which are used to monitor the user's fatigue level and prevent fatigue operation. Such a setting can improve the response speed and accuracy of the humanoid robot to human actions, thereby enhancing the naturalness and fluency of human-computer interaction.

[0063] The multimodal fusion and control platform has high flexibility and adaptability, and can be configured and adjusted according to different task requirements, thereby improving operational efficiency and task completion quality. In some embodiments, the motion sensing wearable glove 4 includes an inertial sensor and a bending sensor; the number of inertial sensors is eleven, and the inertial sensors are respectively distributed at the palm position, the metacarpophalangeal joint of each finger, and the proximal knuckle of each finger; the number of bending sensors is fourteen, and the bending sensors are respectively distributed at the metacarpophalangeal joint of each finger, the proximal knuckle of each finger, and the distal knuckles of the four fingers other than the thumb. The bending sensor is used to sense the angle of the finger bending toward the palm and capture the second bending action of the finger.

[0064] The human signal fusion perception platform profile frame 1 and the dual-arm robot mirror control platform profile frame 2 are both rectangular frames. A data center is provided at the bottom of the human signal fusion perception platform profile frame 1. The data center includes a display 6 and several depth camera industrial computers 7. The display 6 is used for debugging the depth camera industrial computer 7 and visual display of data. The depth camera industrial computer 7 as a terminal is used to connect to the depth camera 5 to collect the collected depth data, and send the collected data synchronously for processing.

[0065] A dual-arm robot mirror control platform controller 10 for controlling a humanoid robotic arm 11 and a humanoid robot dexterous hand 9 is provided on the top of the dual-arm robot mirror control platform profile frame 2. Humanoid robotic arms 11 are provided on opposite sides of the dual-arm robot mirror control platform controller 10. A humanoid robot dexterous hand 9 is provided on the end of the humanoid robotic arm 11 that is not connected to the dual-arm robot mirror control platform controller 10. The humanoid robotic arm 11 includes three shoulder joints, one elbow joint and three wrist joints. Each joint of the humanoid robotic arm 11 adopts a high-precision servo motor and a multi-stage reducer. The humanoid robotic arm 11 adopts a series configuration and operates based on the form of multi-joint connecting rod following.

[0066] The humanoid robot dexterous hand 9 includes six degrees of freedom, among which the thumb structure has two degrees of freedom, including bending degree of freedom and joint degree of freedom, which are used for precise grasping and operation of complex objects. The other four fingers each include one bending degree of freedom. The humanoid robot dexterous hand 9 is used to perform finger bending and grasping actions based on a multi-joint connecting rod follow-up form by integrating high-precision micro-servo motors and connecting rod mechanisms. The humanoid robot dexterous hand 9 has a built-in force tactile sensor.

[0067] The dual-arm robot mirror control platform integrates multimodal sensing technology to receive and fuse the user's hand, forearm and upper arm motion data, as well as other sensor information (such as vision, voice and force feedback data). The platform adopts the same degree of freedom configuration as human arms, including a 7-degree-of-freedom robotic arm design, which can accurately imitate the natural movements of human upper limbs. This highly bionic structure not only supports the precise replication of user movements and realizes the refined remote control operation of the robotic arm, but also generates interactive actions that are adapted to human collaboration needs through algorithm optimization. The platform's robotic arm adopts a serial joint structure, including 3 degrees of freedom at the shoulder (front and back swing, left and right swing and rotation), 1 degree of freedom at the elbow (bending) and 3 degrees of freedom at the wrist (pitch, side swing and rotation), which restores the range of motion and flexibility of the human arm to the greatest extent. At the same time, the humanoid dexterous hand has a five-finger design, of which the thumb has two degrees of freedom, and the other four fingers each have one degree of freedom. It adopts a multi-joint connecting rod follower form, which can efficiently complete fine grasping and complex operations. Each fingertip joint integrates a force tactile sensor, which can sense the corresponding pressure data information during grasping and operation. This design enables the dual-arm robot to achieve natural and high-precision movements in remote operation scenarios, and accurately reproduce human movements in complex environments. In addition, the humanoid structure of the dual-arm robot shows extremely high adaptability in human-computer interaction and collaboration scenarios. By combining the fusion processing of multimodal perception data, the platform can collaborate with humans in a form more similar to human-human interaction, truly realizing human-computer integration. This design is not only suitable for operations in dangerous environments such as medical surgery assistance, nuclear power plants or chemical plants, but also can show strong adaptability and collaboration capabilities in highly complex tasks such as external maintenance of spacecraft.

[0068] In some embodiments, the space size provided by the dual-arm robot mirror control platform profile frame 2 can ensure that the humanoid robot arm 11 and the humanoid robot dexterous hand 9 have sufficient movement space to ensure the safety of movement.

[0069] In some embodiments, a plurality of depth cameras 5 are disposed on the top of the human signal fusion perception platform profile frame 1 and the dual-arm robot mirror control platform controller 10 .

[0070] The depth camera 5 is used to perceive the interactive environment and information of the interactive object (human behavior data and robot depth vision information). In some embodiments, the depth camera 5 is connected to the human signal fusion perception platform profile frame 1 through a slidable device, so that the user's movement can be tracked and captured in time when the user moves. Such a setting can realize multi-angle non-contact collection of user hand movement information, and can avoid occlusion problems. Several depth cameras 5 are driven by motors on the human signal fusion perception platform profile frame 1 and the dual-arm robot mirror control platform controller 10, and the depth camera 5 collection position can be adjusted based on application requirements.

[0071] The dual-arm robot mirror control platform controller 10 can complete the integration, connection, and processing of all components. It is also the center for human interaction. It is mainly used to receive the user's hand, forearm, and upper arm motion information obtained by the human body signal fusion perception platform and the perception data of each sensor, so as to realize the remote control operation of the mechanical arms on both sides of the dual-arm robot mirror control platform controller 10. Refined control. Through the fusion processing of these signals and data, the dual-arm robot mirror control platform controller 10 can accurately imitate the user's actions, or realize interaction and collaboration with the user, and is used to complete designated tasks remotely. The dual-arm robot mirror control platform controller 10 is suitable for various complex environments that require remote operation, including but not limited to medical surgery assistance, operations in dangerous environments (such as nuclear power plants, chemical plants, etc.) and external maintenance of spacecraft.

[0072] In some embodiments, the bottom of the human signal fusion perception platform profile frame 1 and the dual-arm robot mirror control platform profile frame 2 are both provided with universal wheels 8 to facilitate assembly, debugging and movement.

[0073] In some embodiments, a wheeled intelligent sensing chassis is provided at the bottom of the human signal fusion sensing platform profile frame 1 and the dual-arm robot mirror control platform profile frame 2, including inertial sensors and torque sensors, which are used to monitor and adjust the movement state of the platform in real time to ensure the stability and flexibility of the platform in different terrains and environments.

[0074] In some embodiments, the user wears motion sensing gloves 4 and a surface electromyography multi-lead sensing bracelet 3, and moves the upper limbs within the motion space provided by the human body signal fusion sensing platform profile frame 1 to collect relevant signals, electromyography information, and upper limb motion information.

[0075] Through the above design, the present invention provides an efficient, flexible and precise multi-modal fusion and control platform for human-humanoid robot interaction, which is suitable for various application scenarios requiring refined operations and adaptation to complex environments.

[0076] When the multimodal fusion and control platform is used through the above-mentioned embodiments of the present invention, it can be configured in a variety of ways according to the needs of the user. If the user needs to achieve high-precision and complete long-term hand interaction, after entering the motion capture space and wearing the equipment, a series of hand movements are completed in the space. In addition, if the motion action involves the use of other instruments, such as holding other tools, different sensing devices can complement each other, and the depth camera 5 and the motion sensing wearable gloves 4 can complement each other in motion pattern recognition. For example, the acceleration and angular velocity data provided by the motion sensing wearable gloves 4 can supplement the shortcomings of the depth camera 5 under occlusion. At the same time, in a specific movement, the platform can focus on the body parts involved in the action. For example, when performing gestures, you can focus on the position and motion data of the palm to reduce unnecessary data redundancy.

[0077] According to the specific human-computer interaction task requirements, the humanoid robot arm 11 and the humanoid robot dexterous hand 9 can respond to the user's interaction, including the response of the mechanical arms and the response of the five-finger dexterous hand. In addition, in the robot platform, the data information sensed by the depth camera 5 and the tactile information of the five-finger dexterous hand are fed back to the embodied intelligent human-computer interaction algorithm model to realize the closed loop of the process and achieve a real interactive effect.

[0078] In summary, the present invention provides a multimodal fusion and control platform for human-humanoid robot interaction, which improves the quality of interaction between humans and humanoid robots in motion recognition, prediction and tracking, and improves the response speed and accuracy of humanoid robots to human actions, thereby enhancing the naturalness and fluency of human-computer interaction, and ensuring that interaction quality and experience can be provided in various complex environments.

[0079] In a second aspect, the present invention provides a multimodal fusion and control system for human-humanoid robot interaction, the multimodal fusion and control system comprising a multi-sensor fusion module, a spatial coordinate calibration fusion module, an adaptive filtering module and a human-computer interaction module;

[0080] The multi-sensor fusion module is used to collect and fuse data from multiple sensors to obtain multi-sensor fusion data, and send the multi-sensor fusion data to the spatial coordinate calibration fusion module, wherein the data from multiple sensors include surface electromyography data collected by the surface electromyography multi-lead sensing bracelet 3, three-axis acceleration data and three-axis posture data collected by the inertial sensor of the motion sensing wearable glove 4, and curvature data collected by the curvature sensor, and the multi-sensor fusion data includes the fused surface electromyography data and hand motion posture data.

[0081] The spatial coordinate calibration and fusion module is used to correct and synchronize the human behavior data in different coordinate systems collected by the depth camera 5 of the human signal fusion platform profile frame 1, and to perform coordinate correction on the depth visual information collected by the depth camera 5 of the dual-arm robot mirror control platform controller 10 to obtain multi-sensor fusion calibration data, which has the same spatial coordinate system.

[0082] The adaptive filtering module is used to perform adaptive filtering on the human behavior data corrected and synchronized in the multi-sensor fusion calibration data and the pressure data collected by the force tactile sensor in the humanoid robot dexterous hand 9. Sensor devices have their own limitations: the depth camera 5 of the human signal fusion platform profile frame 1 may be affected by the light in the environment, resulting in errors, and the data may be lost or blurred due to insufficient frame rate when the hand moves quickly; the motion sensing wearable gloves 4 are worn on the human hand in the form of wearable devices, and the different hand sizes and wearing methods of different users may cause data deviations. The adaptive filtering technology adaptively adjusts the parameters of the filter and the fusion weights of different data according to the status of different real-time data, and obtains the filtered human behavior data and pressure data, thereby reducing noise and improving data accuracy, significantly improving the tracking performance of the hand in a variety of complex scenarios.

[0083] The human-computer interaction module is used to determine the robot response strategy based on multi-sensor fusion data, filtered human behavior data and pressure data, and based on the robot response strategy, control the humanoid robotic arm 11 and the humanoid robot dexterous hand 9 to perform interactive movements with the user, wherein the robot response strategy includes the position and posture of the humanoid robotic arm 11 and the humanoid robot dexterous hand 9.

[0084] For example, Figure 2 and Figure 3 As shown, the human behavior data and depth visual information, surface electromyography data, finger curvature data and inertial navigation unit information collected by the depth camera 5, the surface electromyography multi-lead sensing bracelet 3, the motion sensing wearable glove 4 and the tactile sensor of the humanoid robot dexterous hand 9 respectively reflect the user's movement, and are subjected to signal preprocessing (such as segmentation), correction and synchronization processing, coordinate correction processing, adaptive filtering processing, etc., to obtain filtered data, and then control the humanoid mechanical arm 11 and the humanoid robot dexterous hand 9 to complete human-computer interaction. In some embodiments, different human-computer interaction intention perception models are selected and constructed according to actual application requirements (such as trajectory planning, human-computer interaction, human-computer collaboration and human-computer integration).

[0085] In some embodiments, in the multi-sensor fusion module,

[0086] The user's hand motion posture is determined according to the following posture estimation formula:

[0087]

[0088] In the formula, q k-1 and q k are the quaternion-based attitude data at the k-1th moment and the kth moment, represents the quaternion multiplication symbol, Δt is the time interval between the k-1th moment and the kth moment, ω k is the angular velocity data at the kth moment;

[0089] According to the following formula, the acceleration data and attitude data are transformed to obtain the linear acceleration in the global coordinate system:

[0090] α global =R(q k )·α

[0091] Among them, α global is the linear acceleration in the global coordinate system, R(q k ) is q k The generated rotation matrix, α is the acceleration data;

[0092] According to the following formula, the position trajectory of the hand movement is determined:

[0093] v k =v k-1 +α global Δt

[0094] p k =p k-1 +v k Δt

[0095] In the formula, v k is the speed of hand movement at the kth moment, p k is the position of the hand movement at the kth moment, v k-1 is the speed of hand movement at the k-1th moment, p k-1 is the position of the hand movement at the k-1th moment;

[0096] According to the following formula, the multi-sensor fusion data is determined:

[0097] θ i =f b (s i )

[0098] P k_fusion =f1(v k ,p k ,θ i )

[0099] Where P k_fusionis the number of hand motion postures in the multi-sensor fusion data at the kth moment, f1() is the fusion transformation function, θ i is the curvature data of the ith finger joint at the kth moment, s i is the resistance change of the i-th bending sensor at the k-th moment, f b () is the change mapping function;

[0100] The surface electromyography data collected by the surface electromyography multi-lead sensor bracelet 3 is expressed as E={E R ,E L}, E={E R ,E L} contains the left hand surface electromyography data and the right hand surface electromyography data, where Represents the data of the i-th lead in the left / right hand surface electromyography data. According to the following formula, the hand muscle state data in the multi-sensor fusion data is determined:

[0101]

[0102] Where M L and M R Represents the hand muscle state data of the left hand and the right hand respectively, f N is the mapping function from surface electromyography data to hand muscle state data.

[0103] In order to ensure that sensors with direction perception functions can perform accurate spatial coordinate calibration, the present invention has developed a stable calibration program. The calibration program can perform high-precision coordinate calibration and change of multiple depth cameras through the spatial posture information and spatial confidence provided, realize the world coordinate system, and improve the overall performance of the invention.

[0104] In some embodiments, in the spatial coordinate calibration fusion module,

[0105] According to the following formula, the corrected and synchronized human behavior data and the coordinate-corrected depth visual information in the multi-sensor fusion calibration data are determined:

[0106]

[0107] Where W is the defined world coordinate system, C i is the coordinate system of the depth camera 5 on the profile frame 1 of the i-th human body signal fusion platform, Indicates that from C i The transformation matrix from the coordinate system to the W coordinate system, Represents human behavior data in the W coordinate system, Indicates that in C i Human behavior data in coordinate system, P WIt represents the human behavior data after correction and synchronization processing in the multi-sensor fusion calibration data in the W coordinate system, f2() is the spatial coordinate fusion function, and n represents the sensor number.

[0108] In some embodiments, the formula for determining the depth visual information after coordinate correction in the multi-sensor fusion calibration data may be the same as the formula for determining the human behavior data after correction and synchronization processing in the multi-sensor fusion calibration data.

[0109] In some embodiments, in the adaptive filtering module,

[0110] P = f3(P W ,P k_fusion )

[0111] F i =f4(M i )

[0112] Where P is the filtered human behavior data, P k_fusion is the hand motion posture data in the multi-sensor fusion data at the kth moment, P W represents the human behavior data after correction and synchronization processing in the multi-sensor fusion calibration data in the W coordinate system, f3() is the fusion function, f4() is the mapping function used to map the pressure data matrix before filtering to the pressure data after filtering, M i is the pressure data matrix M before filtering at the ith finger of the humanoid robot dexterous hand (9) i , F i It represents the pressure data of the ith finger of the humanoid robot dexterous hand (9) obtained by filtering.

[0113] In some embodiments, in the human-computer interaction module, the robot response strategy is determined by generalizing the large model embodied intelligent core algorithm; the angles of each joint of the humanoid robotic arm 11 and the humanoid robot dexterous hand 9 are determined by the robot inverse kinematics solution method.

[0114] In some embodiments, the core algorithm of generalized large model embodied intelligence: based on the generalized large model and embodied intelligence theory, a perception system capable of complex human behavior and environmental data is constructed, and the transfer of generalization capabilities is achieved through the combination of pre-trained large models and real-time environmental perception, so that the algorithm is not only suitable for specific scenarios, but also can cope with complex and changing environments.

[0115] The input of the algorithm mainly includes the following multimodal data, counted as:

[0116] X={x1,x1,x n ,...,x n}

[0117] Among them, x iIt is determined according to the actual task requirements, and X represents the total task. In different actual tasks, the requirements for multimodal data are different. For example, in a certain task, there is no demand for pressure data of the humanoid robot dexterous hand 9. In this scenario, tactile force data can be collected without reducing the data load of the system. Single modality data cannot meet the needs of capturing multiple levels of human behavior. For example, hand activities not only contain hand posture movement information, but also surface electromyography information. Therefore, different tasks are determined according to the actual task requirements.

[0118] Use the pre-trained large model to process the fusion features, denoted as O task = M(X), whose goal is to generate task-related outputs from input features. Different decoding is used for different tasks (such as robot arm motion planning, dexterous hand operation, etc.), dynamically adapting to the data characteristics of different scenarios, and using the understanding and generalization ability of the large model to output the required robot (i.e., humanoid robot arm 11 and humanoid robot dexterous hand 9) response behavior in different scenarios.

[0119] Reinforcement learning framework: With the support of a generalized large model, a reinforcement learning framework is constructed. Based on user interaction data, environmental feedback, and robot responses output by the large model, the robot response strategy is continuously optimized, so that the robot can adjust actions, paths, and language outputs in real time, and achieve dynamic adaptation to complex tasks in practice. The robot strategy is represented by a parameterized model, and the cumulative reward (expected return) is maximized by optimizing the strategy parameters. The purpose of reinforcement learning is to allow the robot to learn when the sample size is insufficient, learn the user's behavior, and achieve the same movement form as the user. It can be continuously trained according to the mechanism of reinforcement learning.

[0120] Robot control algorithm: Based on the robot response strategy given by the generalized large model embodied intelligence core algorithm, the robot inverse kinematics algorithm is used to realize the conversion of control information, and then the adaptive control method is used to realize robot control.

[0121] The robot response strategy given by the generalized large model embodied intelligence core algorithm provides the output of the humanoid robotic arm 11 and the position and posture of the humanoid robotic dexterous hand 9. The angles of each joint of the humanoid robotic arm and the humanoid robotic dexterous hand are solved through robot inverse kinematics for further control.

[0122] According to robot inverse kinematics, for a seven-degree-of-freedom humanoid robot arm, the position of the end effector can be described as:

[0123] T=T1(θ1)·T2(θ2)····T6(θ6)·T7(θ7)

[0124] Among them, T irepresents the transformation matrix of the i-th joint, T is the end position of the robot, θ i Represents the joint angle.

[0125] The joint angle θ can be calculated by numerical solution of inverse kinematics. i The solution is to use the iterative method, which is based on the joint angle error. The iterative expression is as follows:

[0126] θ k+1 =θ k -αJ(θ k ) -1 ·e(θ k )

[0127] Among them, θ k is the current joint angle; J(θ k ) is the Jacobian matrix, which is used to describe the impact of joint changes on the end effector; e(θ k ) is the error between the current end position and the target position; α is the step size factor, which is used to control the iteration step size. Through multiple iterations, we know the error e(θ k ) is small enough to obtain a solution that satisfies accuracy.

[0128] For the anthropomorphic robot dexterous hand 9 , the four fingers except the thumb have only one degree of freedom and can be directly converted; the thumb adopts the same inverse kinematics method as the anthropomorphic robot arm 11 to achieve the conversion of the joint angle.

[0129] The adaptive control algorithm combines the joint angle information provided by inverse kinematics and the feedback signal of the robot to achieve real-time parameter estimation of the robot system model, update the controller parameters online, ensure that the robot can maintain accurate response, and overcome the uncertainty of the system model and external interference. Through the adaptive control algorithm, it is possible to quickly approach the control target at the beginning of control; as the error between input and output gradually decreases, the system parameters and controller parameters change accordingly, approaching the control target with smaller output changes, and achieving a fast and accurate response process. At the same time, in the presence of external interference, the adaptive control algorithm can automatically adjust the parameters to maintain stable control.

[0130] The specific control process is as follows:

[0131] The adaptive control algorithm dynamically estimates the parameters of the control system (such as the inertia of the robot joints, damping coefficients, etc.). The estimation of these parameters usually depends on the input signal, output signal and error signal. The basic form of control parameter update can be expressed as:

[0132]

[0133] Among them, Φ(t) is the parameter estimate of the system; Γ is the adaptive gain matrix; ψ(t) is the regression vector containing the input; e(t) is the difference between the target value and the current output. The parameter estimate of the system will be continuously adjusted according to the current error to learn an accurate system model. At the same time, when subject to external interference, the system parameter estimate is automatically updated to facilitate subsequent stable control.

[0134] The controller of the adaptive control algorithm generates a control signal based on the current parameter estimates and the regression vector. The control signal is in the following form:

[0135] u(t)=Φ T (t)·ψ(t)

[0136] Among them, u(t) is the control signal; Φ(t) is the current parameter estimate; ψ(t) is the regression vector containing the input, reflecting the left and right of the system input. When there is an error between the system output and the target value, the parameter update rule automatically adjusts the control parameters through the feedback mechanism to make the control signal u(t) more accurate, thereby approaching the target output.

[0137] In some embodiments, the depth visual information collected by the depth camera 5 of the dual-arm robot mirror control platform controller 10 and the pressure data collected by the force tactile sensor of the humanoid robot dexterous hand 9 can be sent to the multi-sensor fusion module and the spatial coordinate calibration fusion module in real time for real-time feedback, thereby improving the accuracy of human-computer interaction.

[0138] In summary, the present invention provides a multimodal fusion and control system for interaction between humans and humanoid robots, which improves the quality of interaction between humans and humanoid robots in motion recognition, prediction and tracking, and improves the response speed and accuracy of humanoid robots to human actions, thereby enhancing the naturalness and fluency of human-computer interaction, and ensuring that interaction quality and experience can be provided in various complex environments.

[0139] The above specific embodiments are merely explanations of the present invention and are not limitations of the present invention. After reading this specification, those skilled in the art may make modifications to the embodiments without any creative contribution as needed. However, such modifications are protected by the patent law as long as they are within the scope of the claims of the present invention.

Claims

1. A multimodal fusion and control platform for human-humanoid robot interaction, characterized by: It includes a human body signal fusion perception platform and a dual-arm robot mirror control platform profile frame (2); The human body signal fusion sensing platform comprises a human body signal fusion sensing platform profile frame (1), a surface electromyography multi-lead sensing bracelet (3), and a motion sensing wearable glove (4); The human body signal fusion perception platform profile frame (1) and the dual-arm robot mirror control platform profile frame (2) are both rectangular frames. A data center is provided at the bottom of the human body signal fusion perception platform profile frame (1), and the data center includes a display (6) and a plurality of depth camera industrial computers (7); A dual-arm robot mirror control platform controller (10) is provided on the top of the dual-arm robot mirror control platform profile frame (2); humanoid mechanical arms (11) are provided on opposite sides of the dual-arm robot mirror control platform controller (10); a humanoid mechanical hand (9) is provided at one end of the humanoid mechanical arm (11) that is not connected to the dual-arm robot mirror control platform controller (10); the humanoid mechanical arm (11) comprises three shoulder joints, one elbow joint and three wrist joints; each joint of the humanoid mechanical arm (11) adopts a high-precision servo motor and a multi-stage reducer; the humanoid mechanical arm (11) adopts a series configuration and operates in the form of multi-joint connecting rod follow-up; The humanoid robot dexterous hand (9) comprises six degrees of freedom, wherein the thumb structure comprises two degrees of freedom, including a bending degree of freedom and a joint degree of freedom, which are used for the precise grasping and operation of complex objects, and the remaining four fingers each comprise a bending degree of freedom. The humanoid robot dexterous hand (9) is used to perform finger bending and grasping actions based on a multi-joint connecting rod follow-up form by integrating a high-precision micro-servo motor and a connecting rod mechanism, and the humanoid robot dexterous hand (9) has a built-in force tactile sensor.

2. A multi-modal fusion and control platform for human-humanoid robot interaction as claimed in claim 1, characterized in that: The human signal fusion perception platform profile frame (1) and the dual-arm robot mirror control platform controller (10) are both provided with a plurality of depth cameras (5) on the top.

3. The multimodal fusion and control platform for human-humanoid robot interaction as claimed in claim 1, characterized in that: The motion sensing wearable glove (4) comprises an inertial sensor and a bending sensor, which are used to capture hand movements; There are eleven inertial sensors, which are respectively distributed at the palm position, the metacarpophalangeal joint of each finger, and the proximal phalangeal joint of each finger; There are fourteen bending sensors, which are respectively distributed at the metacarpophalangeal joint of each finger, the proximal knuckle of each finger, and the distal knuckles of the four fingers other than the thumb. The bending sensors are used to sense the bending angle of the finger toward the palm and capture the second bending movement of the finger.

4. The multi-modal fusion and control platform for human-humanoid robot interaction as claimed in claim 2, characterized in that: The depth camera (5) is connected to the human signal fusion perception platform profile frame (1) and the dual-arm robot mirror control platform controller (10) via a slidable device.

5. The multi-modal fusion and control platform for human-humanoid robot interaction as claimed in claim 1, characterized in that: The bottoms of the human body signal fusion perception platform profile frame (1) and the dual-arm robot mirror control platform profile frame (2) are both provided with universal wheels (8).

6. A multimodal fusion and control system for human-humanoid robot interaction, characterized by: The multimodal fusion and control system includes a multi-sensor fusion module, a spatial coordinate calibration fusion module, an adaptive filtering module and a human-computer interaction module; The multi-sensor fusion module is used to collect and fuse data from multiple sensors to obtain multi-sensor fusion data, and send the multi-sensor fusion data to the spatial coordinate calibration fusion module, wherein the data from the multiple sensors include surface electromyography data collected by a surface electromyography multi-lead sensor bracelet (3), three-axis acceleration data and three-axis posture data collected by an inertial sensor of a motion sensing wearable glove (4), and curvature data collected by a curvature sensor, and the multi-sensor fusion data includes fused hand muscle state data and hand motion posture data; The spatial coordinate calibration and fusion module is used to calibrate and synchronously process human behavior data in different coordinate systems collected by the depth camera (5) of the human signal fusion platform profile frame (1), and to perform coordinate calibration on the depth visual information collected by the depth camera (5) of the dual-arm robot mirror control platform controller (10), so as to obtain multi-sensor fusion calibration data, wherein the multi-sensor fusion calibration data has the same spatial coordinate system; The adaptive filtering module is used to perform adaptive filtering on the human behavior data after correction and synchronization processing in the multi-sensor fusion calibration data and the pressure data collected by the built-in force tactile sensor of the humanoid robot dexterous hand (9), so as to obtain filtered human behavior data and pressure data; The human-machine interaction module is used to determine a robot response strategy based on the multi-sensor fusion data, filtered human behavior data and pressure data, and based on the robot response strategy, control the humanoid robotic arm (11) and the humanoid robotic dexterous hand (9) to perform interactive movements with the user, wherein the robot response strategy includes the position and posture of the humanoid robotic arm (11) and the humanoid robotic dexterous hand (9).

7. A multi-modal fusion and control system for human-humanoid robot interaction as claimed in claim 6, characterized in that: In the multi-sensor fusion module, The user's hand motion posture is determined according to the following posture estimation formula: In the formula, q k-1 and q k are the quaternion-based attitude data at the k-1th moment and the kth moment, represents the quaternion multiplication symbol, Δt is the time interval between the k-1th moment and the kth moment, ω k is the angular velocity data at the kth moment; According to the following formula, the acceleration data and attitude data are transformed to obtain the linear acceleration in the global coordinate system: a global =R(q k )·a Among them, α global is the linear acceleration in the global coordinate system, R(q k ) is q k The generated rotation matrix, α is the acceleration data; According to the following formula, the position trajectory of the hand movement is determined: v k =v k-1 +α global Δt what k =p k-1 +v k Δt In the formula, v k is the speed of hand movement at the kth moment, p k is the position of the hand movement at the kth moment, v k-1 is the speed of hand movement at the k-1th moment, p k-1 is the position of the hand movement at the k-1th moment; According to the following formula, the multi-sensor fusion data is determined: i i =f b (s i ) P k_fusion =f1(v k ,p k ,θ i ) Where P k_fusion is the hand motion posture data in the multi-sensor fusion data at the kth moment, f1() is the fusion transformation function, θ i is the curvature data of the ith finger joint at the kth moment, s i is the resistance change of the i-th bending sensor at the k-th moment, f b () is the change mapping function; The surface electromyography data collected by the surface electromyography multi-lead sensor bracelet (3) is expressed as E = {E R ,E L }, E={E R ,E L } contains the left hand surface electromyography data and the right hand surface electromyography data, where Represents the data of the i-th lead in the left / right hand surface electromyography data. According to the following formula, the hand muscle state data in the multi-sensor fusion data is determined: Where M L and M R Represents the hand muscle state data of the left hand and the right hand respectively, f N is the mapping function from surface electromyography data to hand muscle state data.

8. A multi-modal fusion and control system for human-humanoid robot interaction as claimed in claim 6, characterized in that: In the spatial coordinate calibration fusion module, According to the following formula, the corrected and synchronized human behavior data and the coordinate-corrected depth visual information in the multi-sensor fusion calibration data are determined: Where W is the defined world coordinate system, C i is the coordinate system of the depth camera (5) on the i-th human signal fusion platform profile frame (1), Indicates that from C i The transformation matrix from the coordinate system to the W coordinate system, Represents human behavior data in the W coordinate system, Indicates that in C i Human behavior data in coordinate system, P W It represents the human behavior data after correction and synchronization processing in the multi-sensor fusion calibration data in the W coordinate system, f2() is the spatial coordinate fusion function, and n represents the sensor number.

9. A multi-modal fusion and control system for human-humanoid robot interaction as claimed in claim 6, characterized in that: In the adaptive filtering module, P=f3(P W ,P k_fusion ) F i =f4(M i ) Where P is the filtered human behavior data, P k_fusion is the hand motion posture data in the multi-sensor fusion data at the kth moment, P W represents the human behavior data after correction and synchronization processing in the multi-sensor fusion calibration data in the W coordinate system, f3() is the fusion function, f4() is the mapping function used to map the pressure data matrix before filtering to the pressure data after filtering, M i is the pressure data matrix M before filtering at the ith finger of the humanoid robot dexterous hand (9) i , F i It represents the pressure data of the ith finger of the humanoid robot dexterous hand (9) obtained by filtering.

10. A multi-modal fusion and control system for human-humanoid robot interaction as claimed in claim 6, characterized in that: In the human-machine interaction module, the robot response strategy is determined by generalizing the large model embodied intelligent core algorithm; the angles of each joint of the humanoid robot arm (11) and the humanoid robot dexterous hand (9) are determined by the robot inverse kinematics solution method.

Images