Multi-mode wearable humanoid robot data acquisition and remote operation system

Abstract

The invention discloses a multi-mode wearable humanoid robot data acquisition and remote operation system. The system comprises a wearable device which acquires real human body data; the environment sensing equipment collects real environment data; the communication equipment performs multi-mode data transmission; the storage device stores data acquired by the wearable device and the environment sensing device, and performs data writing and data reading; the computing device receives the data collected by the wearable device and the environment sensing device, carries out calculation, maps the human body data into humanoid robot data, and generates a control instruction of the humanoid robot; and the humanoid robot body equipment executes a corresponding action according to the control instruction. Therefore, through the human body posture data, the reasonability of the posture and action of the humanoid robot is improved, the completion rate of similar task learning of the humanoid robot in a complex scene is improved, mature industrial equipment is adopted, the cost is low, the organization and implementation difficulty is small, and the method has remarkable advantages in the aspects of data collection quality and benefits.

Description

Technical Field

[0001] The present invention relates to the technical field of data acquisition, and in particular to a multi-modal wearable humanoid robot data acquisition and teleoperation system. Background Art

[0002] With the development of science and technology, the application of robots is becoming more and more extensive. In some complex and dangerous environments, robots that can replace humans to enter the scene and operate the target are needed. For this reason, humanoid robots have come into being. Nowadays, a large number of humanoid robots have been developed. They are expected to be as similar to humans as possible and complete robot work by interacting with objects in the environment, such as people or objects in the environment.

[0003] However, due to the complexity and diversity of the objects that humanoid robots interact with in the environment, as well as the lack of multimodal data on humanoid robots, humanoid robots are unable to fully perform automated operations in complex scenes with emergencies. Summary of the invention

[0004] The present invention provides a multi-modal wearable humanoid robot data acquisition and teleoperation system to solve the problems of insufficient operating environment data and insufficient ability to solve emergencies through real-time teleoperation by humans.

[0005] The embodiment of the present invention provides a multi-modal wearable humanoid robot data acquisition and teleoperation system, comprising:

[0006] Wearable devices, used to collect real human body data;

[0007] Environmental sensing equipment, used to collect real environment data;

[0008] Communication equipment for multi-modal data transmission;

[0009] A storage device, used to store data collected by the wearable device and the environment sensing device, and to write and read data;

[0010] A computing device, used to receive data collected by the wearable device and the environment sensing device, perform calculations, map human body data to humanoid robot data, and generate control instructions for the humanoid robot;

[0011] The humanoid robot main body device is used to perform corresponding actions according to the control instructions.

[0012] Optionally, in one embodiment of the present invention, the wearable device includes: a VR head display device, a gesture sensor device, a data glove device and a depth camera device.

[0013] Optionally, in one embodiment of the present invention, the environmental perception device includes: a color depth camera, an infrared temperature measurement camera, a sound source localization device, and a laser ranging device, which are used to collect environmental information such as images, sounds, temperature, and point clouds.

[0014] Optionally, in one embodiment of the present invention, the communication device includes a multi-stage router or switch.

[0015] Optionally, in one embodiment of the present invention, the humanoid robot body device includes: a head with at least 2 degrees of freedom, 2 robotic arms with at least 6 degrees of freedom, 2 legs with at least 6 degrees of freedom, a torso with at least 2 degrees of freedom and 2 dexterous hands with at least 6 degrees of freedom or a gripper with at least 1 degree of freedom.

[0016] Optionally, in one embodiment of the present invention, the posture sensor device is composed of a plurality of wearable MEMS sensors, and real-time data of each wearing position of the human body is obtained through the plurality of wearable MEMS sensors. The posture of the wearing position is preliminarily solved and fused by the computing device to obtain the posture of each wearing position. The posture quaternions of the human torso, arms and hands, legs and feet are obtained through data conversion, and the posture data is mapped to obtain real-time control instructions for the humanoid robotic arms and lower limbs of the humanoid robot.

[0017] Optionally, in one embodiment of the present invention, the data of the VR head display device includes:

[0018] Human head movement inertial measurement data, human eye tracker measurement data, human hand posture data, human joystick operation data, and image data collected by the VR device camera;

[0019] The data collected by the attitude sensor device include: gyroscope data, acceleration data and magnetometer data;

[0020] The data collected by the multi-data glove device include: finger curvature data, acceleration sensor data, gyroscope data and fingertip pressure data.

[0021] Optionally, in one embodiment of the present invention, the data collected by the color depth camera includes: color image data, depth image data, and point cloud data based on alignment of the color and depth images.

[0022] Optionally, in one embodiment of the present invention, the computing device is specifically used to calibrate and filter the multimodal data collected by the wearable device and the environmental sensing device, establish a human kinematic model, use the collected multimodal data to analyze human motion, convert the human motion data into humanoid robot data, use the humanoid robot data to generate control instructions for the humanoid robot, send the control instructions to the humanoid robot, receive joint feedback data from the humanoid robot, and adjust the humanoid robot data according to the joint feedback data.

[0023] The multi-modal wearable humanoid robot data acquisition and teleoperation system of the embodiment of the present invention has the following beneficial effects:

[0024] 1. Through the detailed imitation learning of human posture data, especially hand data, the rationality of humanoid robot posture and movement can be improved, and the completion rate of humanoid robot learning similar tasks in complex scenes can be improved.

[0025] 2. It uses mature industrial equipment with low cost and easy organization and implementation, and has significant advantages in terms of the quality and efficiency of data collection.

[0026] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0028] Figure 1 A schematic diagram of the structure of a multi-modal wearable humanoid robot data acquisition and teleoperation system provided according to an embodiment of the present invention;

[0029] Figure 2 A network connection topology diagram of a humanoid robot data acquisition and teleoperation system according to an embodiment of the present invention;

[0030] Figure 3 Schematic diagram of a human body posture calculation algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0032] like Figure 1As shown, the multi-modal wearable humanoid robot data acquisition and teleoperation system includes:

[0033] Wearable devices, used to collect real human body data;

[0034] Environmental sensing equipment, used to collect real environment data;

[0035] Communication equipment for multi-modal data transmission;

[0036] Storage device, used to store data collected by wearable devices and environmental sensing devices, and to write and read data;

[0037] A computing device, used to receive data collected by the wearable device and the environmental sensing device, perform calculations, map the human body data to humanoid robot data, and generate control instructions for the humanoid robot;

[0038] The humanoid robot body device is used to perform corresponding actions according to control instructions.

[0039] In an embodiment of the present invention, Figure 2 As shown, the wearable device is used to complete the operator's posture data collection and environmental information feedback display data display device, including VR head display equipment (including somatosensory controller), posture sensor equipment, multimodal data glove equipment (including pressure, temperature and humidity, bending, posture and other sensors) and depth camera equipment.

[0040] VR devices include VR headsets and somatosensory remote controls. Somatosensory remote control data is transmitted to computing devices via WiFi TCP / IP.

[0041] Optionally, the motion sensing remote control data is transmitted to the computing device via USB or network connection.

[0042] like Figure 2 As shown, the multimodal data glove integrates a microelectronic gyroscope, a thin film curvature sensor, a thin film pressure sensor, an infrared temperature sensor, and a humidity sensor. In addition, the microelectronic gyroscope is read through the I2C transmission protocol, and the thin film curvature sensor and the pressure or infrared sensor are read using the ADC to serial port protocol, and the data glove transmits the data to the computing platform wirelessly through WiFi UDP / IP.

[0043] Optionally, the somatosensory remote control data is transmitted to the computing device via a wireless protocol network connection such as Bluetooth or StarFlash.

[0044] In an embodiment of the present invention, environmental perception devices include but are not limited to: color depth cameras, infrared temperature measurement cameras, sound source localization devices, and laser ranging devices, which are used to collect environmental information such as images, sounds, temperatures, and point clouds.

[0045] In an embodiment of the present invention, Figure 2 As shown, the communication equipment is used to complete data communication, sensor information acquisition and data recording, including multi-level routing or switches. The present invention uses WIFI to complete wireless communication, and uses UDP / IP protocol to complete data encapsulation, addressing and transmission, increase the reliability of the platform communication system, and take into account both wired and wireless communication requirements.

[0046] In an embodiment of the present invention, the humanoid robot body device is used to receive movement and control instructions and perform actions such as grasping, placing, carrying, wiping, pulling, etc., and is a humanoid robot body, including: a head with at least 2 degrees of freedom, 2 robotic arms with at least 6 degrees of freedom, 2 legs with at least 6 degrees of freedom, 1 torso with at least 2 degrees of freedom and 2 dexterous hands with at least 6 degrees of freedom or a gripper with at least 1 degree of freedom.

[0047] In an embodiment of the present invention, the posture sensor device is composed of multiple wearable MEMS sensors, and real-time data of each wearing position of the human body is obtained through the multiple wearable MEMS sensors. The posture of the wearing position is preliminarily solved and integrated by a computing device to obtain the posture of each wearing position. The quaternion of the posture of the human body trunk, upper and lower arms and hands, upper and lower legs, and feet is obtained through data conversion, and the posture data is mapped to obtain the real-time control instructions of the humanoid robotic arms and lower limbs of the humanoid robot.

[0048] like Figure 3 As shown, the attitude quaternion data can be represented by the Euler angle method, the direction cosine method and the quaternion method. Among them, the direction cosine method and the quaternion method are often used in the attitude research of integrating the angular velocity, and the quaternion method is also often used in the fusion algorithm based on acceleration and angular velocity. The quaternion method only needs to solve four elements. Compared with the direction cosine method, the amount of calculation is relatively small, and the attitude analysis work of the full angle can be completed, so it is widely used in the fields of navigation and attitude tracking. Therefore, the present invention converts the human motion data collected by multiple IMU sensors into attitude motion parameters described by quaternions. Then, the real-time measurement information of multiple IMUs is comprehensively used, and selective superposition is performed based on their different characteristics, and data fusion is realized by using the hardware extended Kalman filter to obtain attitude data with less noise.

[0049] In an embodiment of the present invention, a computing device includes a memory and a processor, a computer program is stored in the memory, and a computing function is realized when the processor executes the computer program. The computing device is specifically used to calibrate and filter the multimodal data collected by the wearable device and the environmental sensing device, establish a human kinematics model, analyze human motion using the collected multimodal data, convert the human motion data into humanoid robot data, generate control instructions for the humanoid robot using the humanoid robot data, send the control instructions to the humanoid robot, receive joint feedback data of the humanoid robot, and adjust the humanoid robot data according to the joint feedback data.

[0050] The present invention combines the basic principles of robot motion kinematics, dynamics and control to build a system architecture supported by hardware, sensors, actuators and communication networks, to realize the control loop of "data acquisition - data storage and calculation - control generation - feedback data" and remote control of the movable platform, so as to achieve real-time control by humans in the loop.

[0051] In an embodiment of the present invention, the storage device includes a removable hard disk and a network storage system for storing multimodal data. The data format of multimodal data collection includes VR head display device data, human posture data, data glove data, color depth camera data, sound source localization data, and humanoid robot body data.

[0052] The present invention adopts multi-modality to complete the human motion data collection and constructs a mapping matrix between sensor posture information and human posture information. It further optimizes the number and layout of sensors, reduces the matrix dimension and calculation complexity, and realizes real-time solution of the matrix equation on the premise that the matrix equation has a unique solution.

[0053] In an embodiment of the present invention, the data of the VR head display device includes:

[0054] Human head movement inertial measurement data, human eye tracker measurement data, human hand posture data, human joystick operation data, and image data collected by the VR device camera;

[0055] The data collected by the attitude sensor device include: gyroscope data, acceleration data and magnetometer data;

[0056] The data collected by the multi-data glove device include: finger curvature data, acceleration sensor data, gyroscope data and fingertip pressure data.

[0057] In an embodiment of the present invention, the data collected by the color depth camera includes: color image data, depth image data, and point cloud data based on the alignment of the color and depth images.

[0058] In an embodiment of the present invention, the human posture perception system can be replaced by an optical motion capture system based on marker points and an optical capture system based on non-marker points. The optical human motion capture system based on marker points has higher accuracy, but will increase the corresponding site restrictions and equipment costs. The optical capture system based on non-marker points is not as accurate as the current solution, but will also increase the corresponding site restrictions.

[0059] According to the multimodal wearable humanoid robot data acquisition and remote control system proposed in the embodiment of the present invention, based on the "Human-in-loop" human-in-loop concept, a multimodal wearable data acquisition and remote control system is proposed, which simultaneously solves the problem of insufficient working environment data and the problem of insufficient ability to solve emergencies through real-time human remote control.

[0060] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0061] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

Claims

1. A multi-modal wearable humanoid robot data acquisition and teleoperation system, characterized in that: include: Wearable devices, used to collect real human body data; Environmental sensing equipment, used to collect real environment data; Communication equipment for multi-modal data transmission; A storage device, used to store data collected by the wearable device and the environment sensing device, and to write and read data; A computing device, used to receive data collected by the wearable device and the environment sensing device, perform calculations, map human body data to humanoid robot data, and generate control instructions for the humanoid robot; The humanoid robot main body device is used to perform corresponding actions according to the control instructions.

2. The system according to claim 1, characterized in that The wearable device includes: a VR head display device, a posture sensor device, a data glove device and a depth camera device.

3. The system according to claim 1, characterized in that The environmental perception equipment includes: a color depth camera, an infrared temperature measurement camera, a sound source localization device, and a laser ranging device, which are used to collect environmental information such as images, sounds, temperature, and point clouds.

4. The system according to claim 1, characterized in that The communication equipment includes multi-stage routers or switches.

5. The system according to claim 1, characterized in that The humanoid robot body device includes: a head with at least 2 degrees of freedom, two mechanical arms with at least 6 degrees of freedom, two legs with at least 6 degrees of freedom, a torso with at least 2 degrees of freedom and two dexterous hands with at least 6 degrees of freedom or a gripper with at least 1 degree of freedom.

6. The system according to claim 2, characterized in that The posture sensor device is composed of multiple wearable MEMS sensors, which obtain real-time data of various wearing positions of the human body through the multiple wearable MEMS sensors, and use the computing device to preliminarily solve and fuse the posture of the wearing position to obtain the posture of each wearing position. The quaternions of the posture of the human body trunk, arms and hands, legs and feet are obtained through data conversion, and the posture data are mapped to obtain real-time control instructions for the humanoid mechanical arms and lower limbs of the humanoid robot.

7. The system according to claim 2, characterized in that Data for VR headsets includes: Human head movement inertial measurement data, human eye tracker measurement data, human hand posture data, human joystick operation data, and image data collected by the VR device camera; The data collected by the attitude sensor device include: gyroscope data, acceleration data and magnetometer data; The data collected by the multi-data glove device include: finger curvature data, acceleration sensor data, gyroscope data and fingertip pressure data.

8. The system according to claim 3, characterized in that The data collected by the color depth camera includes: color image data, depth image data, and point cloud data based on the alignment of color and depth images.

9. The system according to claim 1, characterized in that The computing device is specifically used to calibrate and filter the multimodal data collected by the wearable device and the environmental sensing device, establish a human kinematics model, use the collected multimodal data to analyze human motion, convert the human motion data into humanoid robot data, use the humanoid robot data to generate control instructions for the humanoid robot, send the control instructions to the humanoid robot, receive joint feedback data from the humanoid robot, and adjust the humanoid robot data according to the joint feedback data.

Images