Multimodal wearable humanoid robot data acquisition and teleoperation system

Abstract

The application discloses a multi-modal wearable humanoid robot data collection and remote operation system, which comprises: a wearable device collecting real human body data; an environment sensing device collecting real environment data; a communication device transmitting multi-modal data; a storage device storing the data collected by the wearable device and the environment sensing device, and performing data writing and data reading; a computing device receiving the data collected by the wearable device and the environment sensing device, and performing calculation, mapping the human body data into humanoid robot data, and generating control instructions of the humanoid robot; and a humanoid robot body device performing corresponding actions according to the control instructions. Thus, the rationality of the posture and action of the humanoid robot is improved through the human body posture data, the humanoid robot's learning similar task completion rate in a complex scene is improved, mature industrialized equipment is adopted, the cost is low, the organization and implementation difficulty is small, and the quality and efficiency of data collection have remarkable advantages.

Description

Technical Field

[0001] This invention relates to the field of data acquisition technology, and in particular to a multimodal wearable humanoid robot data acquisition and remote control system. Background Technology

[0002] With the development of technology, the application of robots is becoming increasingly widespread. In some complex and dangerous environments, robots can replace humans in entering the scene to operate on targets, thus giving rise to humanoid robots. Today, a large number of humanoid robots have been developed, and they are expected to be as similar to humans as possible, completing robotic tasks by interacting with objects in the environment, such as people or objects in the environment.

[0003] However, due to the complexity and diversity of humanoid robots' interactions with objects in the environment, coupled with the lack of multimodal data for humanoid robots, humanoid robots cannot fully perform automated operations in complex situations with unexpected events. Summary of the Invention

[0004] This invention provides a multimodal wearable humanoid robot data acquisition and remote control system to solve the problems of insufficient data in the working environment and insufficient ability to solve emergencies through real-time human remote control.

[0005] This invention provides a multimodal wearable humanoid robot data acquisition and remote control system, comprising:

[0006] Wearable devices used to collect real human body data;

[0007] Environmental sensing devices are used to collect real-world environmental data;

[0008] Communication equipment used for multimodal data transmission;

[0009] A storage device is used to store data collected by the wearable device and the environmental sensing device, and to perform data writing and data reading.

[0010] A computing device is used to receive data collected by the wearable device and the environmental sensing device, perform calculations, map human body data to humanoid robot data, and generate control commands for the humanoid robot.

[0011] A humanoid robot body device, used to execute corresponding actions according to the control commands.

[0012] Optionally, in one embodiment of the present invention, the wearable device includes: a VR headset, a posture sensor, a data glove, and a depth camera.

[0013] Optionally, in one embodiment of the present invention, the environmental sensing device includes: a color depth camera, an infrared temperature measurement camera, a sound source localization device, and a laser ranging device, used to collect environmental information such as images, sound, temperature, and point clouds.

[0014] Optionally, in one embodiment of the present invention, the communication device includes a multi-level router or switch.

[0015] Optionally, in one embodiment of the present invention, the humanoid robot body device includes: a head with at least 2 degrees of freedom, two robotic arms with at least 6 degrees of freedom, two legs with at least 6 degrees of freedom, a torso with at least 2 degrees of freedom, and two dexterous hands with at least 6 degrees of freedom or a gripper with at least 1 degree of freedom.

[0016] Optionally, in one embodiment of the present invention, the posture sensor device is composed of multiple wearable MEMS sensors. Real-time data of various wearable positions of the human body are acquired through multiple wearable MEMS sensors. The posture of each wearable position is initially calculated and fused using the computing device. The posture of each wearable position is obtained by converting the data to obtain the posture quaternions of the human torso, upper and lower arms and hands, lower and upper legs and feet. The posture data is then mapped to obtain the real-time control commands of the humanoid robot's humanoid robotic arm and lower limbs.

[0017] Optionally, in one embodiment of the present invention, the data of the VR headset device includes:

[0018] Human head motion inertial measurement data, human eye tracker measurement data, human hand posture data, human joystick operation data, and image data collected by VR device cameras;

[0019] The data collected by the attitude sensor device includes: gyroscope data, acceleration data, and magnetometer data;

[0020] The data collected by the multi-data glove device includes: finger flexion data, accelerometer data, gyroscope data, and fingertip pressure data.

[0021] Optionally, in one embodiment of the present invention, the data acquired by the color depth camera includes: color image data, depth image data, and point cloud data based on the alignment of color and depth maps.

[0022] Optionally, in one embodiment of the present invention, the computing device is specifically used to calibrate and filter the multimodal data collected by the wearable device and the environmental sensing device, establish a human kinematic model, perform human motion analysis using the collected multimodal data, convert the human motion data into humanoid robot data, generate control commands for the humanoid robot using the humanoid robot data, send the control commands to the humanoid robot, receive joint feedback data from the humanoid robot, and adjust the humanoid robot data according to the joint feedback data.

[0023] The multimodal wearable humanoid robot data acquisition and remote control system of this invention has the following beneficial effects:

[0024] 1. By meticulously imitating and learning human posture data, especially hand data, the rationality of humanoid robot postures and movements can be improved, thereby increasing the completion rate of humanoid robots in learning similar tasks in complex scenarios.

[0025] 2. It adopts mature industrial equipment, which is low-cost and easy to organize and implement, and has significant advantages in terms of the quality and efficiency of data collection.

[0026] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0027] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0028] Figure 1 This is a schematic diagram of a multimodal wearable humanoid robot data acquisition and remote control system according to an embodiment of the present invention.

[0029] Figure 2 This is a network connection topology diagram of the humanoid robot data acquisition and remote control system according to an embodiment of the present invention;

[0030] Figure 3 This is a schematic diagram of the human posture calculation algorithm according to an embodiment of the present invention. Detailed Implementation

[0031] Embodiments of the present invention are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0032] like Figure 1As shown, this multimodal wearable humanoid robot data acquisition and remote control system includes:

[0033] Wearable devices used to collect real human body data;

[0034] Environmental sensing devices are used to collect real-world environmental data;

[0035] Communication equipment used for multimodal data transmission;

[0036] Storage devices are used to store data collected by wearable devices and environmental sensing devices, and to write and read data.

[0037] The computing device is used to receive data collected by wearable devices and environmental sensing devices, perform calculations, map human body data to humanoid robot data, and generate control commands for the humanoid robot.

[0038] The humanoid robot body is used to execute corresponding actions according to control commands.

[0039] In embodiments of the present invention, such as Figure 2 As shown, wearable devices are devices used to collect operator posture data and display environmental information feedback. They include VR headsets (with motion controllers), posture sensors, multimodal data gloves (containing various sensors such as pressure, temperature, humidity, bending, and posture sensors), and depth cameras.

[0040] VR devices include VR headsets and motion-sensing remote controls. Motion-sensing remote control data is transmitted to the computing device via Wi-Fi TCP / IP.

[0041] Optionally, motion-sensing remote control data can be transmitted to a computing device via USB or network connection.

[0042] like Figure 2 As shown, the multimodal data glove integrates a microelectronic gyroscope, a thin-film bending sensor, a thin-film pressure sensor, an infrared temperature sensor, and a humidity sensor. In addition, the microelectronic gyroscope is transmitted via the I2C protocol, and the thin-film bending sensor and pressure or infrared sensor are read using the ADC to serial port protocol. The data is then wirelessly transmitted to the computing platform via Wi-Fi UDP / IP by the data glove.

[0043] Optionally, motion-sensing remote control data can be transmitted to a computing device via wireless network connections such as Bluetooth or StarFlash.

[0044] In embodiments of the present invention, the environmental sensing device includes, but is not limited to, a color depth camera, an infrared temperature measurement camera, a sound source localization device, and a laser ranging device, used to collect environmental information such as images, sounds, temperature, and point clouds.

[0045] In embodiments of the present invention, such as Figure 2 As shown, the communication equipment is used to complete data communication, sensor information acquisition, and data recording, and includes multi-level routers or switches. This invention uses WIFI to complete wireless communication and UDP / IP protocol to complete data encapsulation, addressing, and transmission, increasing the reliability of the platform communication system and accommodating both wired and wireless communication needs.

[0046] In embodiments of the present invention, the humanoid robot body device is used to receive movement and control commands and perform actions such as grasping, placing, carrying, wiping, and pulling. The humanoid robot body includes: a head with at least 2 degrees of freedom, two robotic arms with at least 6 degrees of freedom, two legs with at least 6 degrees of freedom, a torso with at least 2 degrees of freedom, and two dexterous hands with at least 6 degrees of freedom or a gripper with at least 1 degree of freedom.

[0047] In an embodiment of the present invention, the attitude sensor device consists of multiple wearable MEMS sensors. Real-time data of various wearable positions of the human body are acquired through multiple wearable MEMS sensors. The attitude of each wearable position is initially calculated and fused by a computing device to obtain the attitude of each wearable position. The attitude data is mapped to obtain the quaternions of the human torso, upper and lower arms and hands, lower and upper legs and feet through data conversion to obtain the real-time control commands of the humanoid robot's humanoid robotic arm and lower limbs.

[0048] like Figure 3 As shown, attitude quaternion data can be represented using Euler angles, direction cosine, and quaternions. Among these, direction cosine and quaternion methods are commonly used in attitude studies involving the integration of angular velocities, with quaternions also frequently used in fusion algorithms based on acceleration and angular velocity. Quaternions require solving only four elements, resulting in less computation compared to direction cosine methods, and can perform attitude analysis across all angles, making them widely used in navigation and attitude tracking. Therefore, this invention transforms human motion data acquired from multiple IMU sensors into attitude motion parameters described by quaternions. Then, it integrates real-time measurements from multiple IMUs, selectively superimposing them based on their different characteristics, and uses a hardware extended Kalman filter to achieve data fusion, thereby obtaining attitude data with lower noise.

[0049] In embodiments of the present invention, the computing device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform computational functions. Specifically, the computing device is used to calibrate and filter multimodal data collected by wearable devices and environmental sensing devices, establish a human kinematic model, analyze human motion using the collected multimodal data, convert human motion data into humanoid robot data, generate control commands for the humanoid robot using the humanoid robot data, send the control commands to the humanoid robot, receive joint feedback data from the humanoid robot, and adjust the humanoid robot data based on the joint feedback data.

[0050] This invention combines the basic principles of robot kinematics, dynamics, and control to build a system architecture supported by hardware, sensors, actuators, and communication networks. It realizes a control loop of "data acquisition - data storage and computation - control generation - feedback data" and remote control of a mobile platform, achieving real-time human-in-the-loop control.

[0051] In embodiments of the present invention, the storage device includes a removable hard disk and a network storage system for storing multimodal data. The data formats acquired through multimodal data acquisition include VR headset data, human posture data, data glove data, color depth camera data, sound source localization data, and humanoid robot body data.

[0052] This invention employs multimodal methods to acquire human motion data and constructs a mapping matrix between sensor posture information and human posture information. Furthermore, based on the premise that the matrix equation has a unique solution, it optimizes the number and layout of sensors, reduces the matrix dimension and computational complexity, and enables real-time solving of the matrix equation.

[0053] In embodiments of the present invention, the data of the VR headset device includes:

[0054] Human head motion inertial measurement data, human eye tracker measurement data, human hand posture data, human joystick operation data, and image data collected by VR device cameras;

[0055] The data collected by the attitude sensor device includes: gyroscope data, acceleration data, and magnetometer data;

[0056] The data collected by the multi-data glove device includes: finger flexion data, accelerometer data, gyroscope data, and fingertip pressure data.

[0057] In embodiments of the present invention, the data acquired by the color depth camera includes: color image data, depth image data, and point cloud data based on the alignment of color and depth maps.

[0058] In embodiments of the present invention, the human posture perception system can be replaced by a marker-based optical motion capture system and a markerless optical capture system. The marker-based optical motion capture system offers higher accuracy but increases space limitations and equipment costs. The markerless optical capture system is less accurate than the current solution but also incurs space limitations.

[0059] The multimodal wearable humanoid robot data acquisition and remote control system proposed in this embodiment of the invention is based on the "Human-in-loop" concept. It solves the problems of insufficient data in the working environment and insufficient ability to solve emergencies through real-time human remote control.

[0060] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0061] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A multimodal wearable humanoid robot data acquisition and remote control system, characterized in that, include: Wearable devices used to collect real human body data; The wearable devices include: VR headsets, posture sensor devices, data gloves, and depth cameras; The data from VR headsets includes: human head motion inertial measurement data, human eye tracker measurement data, human hand posture data, human joystick operation data, and image data captured by the VR device's camera; The data collected by the attitude sensor device includes: gyroscope data, acceleration data, and magnetometer data; The data collected by the multi-data glove device includes: finger flexion data, accelerometer data, gyroscope data, and fingertip pressure data; Environmental sensing devices are used to collect real-world environmental data; Communication equipment used for multimodal data transmission; A storage device is used to store data collected by the wearable device and the environmental sensing device, and to perform data writing and data reading. A computing device is used to receive data collected by the wearable device and the environmental sensing device, perform calculations, map human body data to humanoid robot data, and generate control commands for the humanoid robot. A humanoid robot body device, used to execute corresponding actions according to the control commands; The attitude sensor device consists of multiple wearable MEMS sensors. Real-time data of various wearable positions of the human body is acquired through multiple wearable MEMS sensors. The computing device performs preliminary calculation and fusion of the attitude of each wearable position to obtain the attitude of each wearable position. Through data conversion, the attitude quaternions of the human torso, upper and lower arms and hands, lower and upper legs and feet are obtained. The attitude data is then mapped to obtain the real-time control commands of the humanoid robot's humanoid robotic arm and lower limbs.

2. The system according to claim 1, characterized in that, The environmental sensing equipment includes: a color depth camera, an infrared temperature measurement camera, a sound source localization device, and a laser ranging device, used to collect environmental information such as images, sound, temperature, and point clouds.

3. The system according to claim 1, characterized in that, The communication equipment includes multiple levels of routers or switches.

4. The system according to claim 1, characterized in that, The humanoid robot body device includes: a head with at least 2 degrees of freedom, two robotic arms with at least 6 degrees of freedom, two legs with at least 6 degrees of freedom, a torso with at least 2 degrees of freedom, and two dexterous hands with at least 6 degrees of freedom or a gripper with at least 1 degree of freedom.

5. The system according to claim 2, characterized in that, The data acquired by the color depth camera includes: color image data, depth image data, and point cloud data based on the alignment of color and depth maps.

6. The system according to claim 1, characterized in that, The computing device is specifically used to calibrate and filter the multimodal data collected by the wearable device and the environmental sensing device, establish a human kinematic model, analyze human motion using the collected multimodal data, convert human motion data into humanoid robot data, generate control commands for the humanoid robot using the humanoid robot data, send the control commands to the humanoid robot, receive joint feedback data from the humanoid robot, and adjust the humanoid robot data according to the joint feedback data.

Images