System for collecting multi-mode body data

Abstract

The invention relates to the technical field of body data acquisition, in particular to a system for acquiring multi-mode body data. The system comprises a data acquisition helmet, data acquisition gloves, data acquisition dynamic capture clothes and data acquisition shoes, a user sequentially wears a data acquisition helmet, data acquisition gloves, data acquisition dynamic capture clothes and data acquisition shoes, and then data information is acquired; collected data information is uploaded to an upper computer through wired connection. The system can be worn by a real operator and body data acquisition is carried out, so that body intelligent data which is flexible in operation and high in success rate can be obtained; and meanwhile, the acquisition system is provided with a plurality of sensors of different types, so that body data of different modes can be acquired.

Description

Technical Field

[0001] The present invention relates to the technical field of embodied data collection, and in particular to a system for collecting multimodal embodied data. Background Art

[0002] Embodied intelligence is one of the most popular application directions in the field of AI in recent years. It combines the research results of robotics and large language models. The core goal is to obtain embodied intelligent entities with real-time interaction capabilities, autonomous movement capabilities, and task planning capabilities through data learning. At the current stage, one of the most critical elements of embodied intelligence is high-quality, large-capacity, multi-scenario embodied data, which is used for several key tasks such as embodied intelligence model training, model fine-tuning, and model migration;

[0003] There are three main types of existing embodied intelligence data: the first type is the physical robot operation data, which is convenient for direct use in model learning and model deployment, and is usually collected by cameras and built-in sensors of the robot; the second type is the simulation robot data, which is usually deployed in a software simulation system and collected by a computer software program; the third type is motion capture data, which is usually collected by a third-person motion capture system or a first-person motion capture system composed of a small number of sensors.

[0004] For the operation data of physical robots, there are shortcomings such as limited operation flexibility and difficulty in scene migration, and it is impossible to efficiently collect data on successful robot operations in multiple scenes; for the data of simulated robots, there are shortcomings such as low data authenticity and limited migration capabilities. Models using simulated data are difficult to deploy and apply in real robot scenes; for existing motion capture data, there are characteristics such as limited collection methods, limited data perspectives and data modalities, and it is impossible to flexibly collect effective data in multiple scenes, and it is impossible to obtain multimodal data that is valuable for training embodied intelligent models. Therefore, how to design a system that is convenient for collecting motion capture data has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of the present invention is to address the problems existing in the background technology and propose a system for collecting multimodal embodied data. When the user wears the helmet, gloves, motion capture suit and shoes of the system, the different sensors arranged thereon can flexibly and quickly perform effective data collection in multiple scenes. The data types are diverse and the data collection is convenient.

[0006] The technical solution of the present invention is a system for collecting multimodal embodied data, including a data collection helmet, data collection gloves, data collection motion capture suit and data collection shoes;

[0007] The data collection helmet is used to collect color visual images, depth visual images, sound voice information and / or fisheye lens images;

[0008] Two data collection gloves are provided to collect the motion status of finger joints and the contact information between fingers and objects;

[0009] Data collection motion capture suit is used to collect the whole body posture information of the human body;

[0010] The data collection shoes are used to collect the position information and pressure information of the sole of the foot;

[0011] The user puts on the data collection helmet, data collection gloves, data collection motion capture suit, and data collection shoes in sequence to collect data information; the collected data information is uploaded to the host computer through a wired connection.

[0012] In an optional embodiment, the data acquisition helmet includes a helmet body, a first bracket, a second bracket, a sliding assembly, and a counterweight assembly;

[0013] A first bracket is provided, and is located at the front of the helmet body; a RealSense RGBD camera is installed on the first bracket, and is used to capture image information of the front of the helmet body; the first bracket is a three-axis gimbal;

[0014] The sliding components are symmetrically arranged on the left and right sides of the helmet body; the second bracket is installed on the sliding component and slides relatively along the sliding component; the second brackets located on both sides are respectively provided with RGB cameras or ultra-wide-angle cameras;

[0015] The counterweight assembly is fixed at the rear end of the helmet body and is used to level the center of gravity of the entire helmet body so that it is located at the geometric center of the helmet body; the counterweight assembly is integrated with a power supply module to supply power to the sampling device; the power supply module is electrically connected to each electrical device through a wire;

[0016] A matching strap for fixing is arranged on the helmet body.

[0017] In an optional embodiment, the helmet body includes an outer hard protective layer and an inner lining layer;

[0018] The first bracket, the sliding assembly and the counterweight assembly are all fixed on the hard protective layer;

[0019] The inner lining layer is a flexible cushion layer in direct contact with the head; the hard protective layer is provided with a plurality of strip-shaped slots;

[0020] The inner lining layer is provided with a plurality of openings for ventilation; the positions of the openings correspond to the positions of the grooves.

[0021] In an optional embodiment, an ultra-wide-angle camera or a microphone array integrated with an IMU posture sensor is respectively arranged on the second brackets located on both sides.

[0022] Preferably, the second bracket is a serpentine bracket.

[0023] In an optional embodiment, the second bracket includes a spherical universal joint, a connecting rod and a fixer; the universal joint is a universal joint with a damper, and the angle is kept fixed after the position is manually adjusted;

[0024] One end of the connecting rod is fixed on the universal joint, and the other end is provided with a fixer for fixed connection with the sampling device.

[0025] In an optional embodiment, the sliding assembly includes a base, a guide bar and a connecting portion;

[0026] The base is formed integrally with the hard protective layer as a base material; the guide strip is arranged on the base, and the connecting parts are arranged in pairs and are both installed on the guide strip and slide relative to the guide strip;

[0027] The connecting part is a cylindrical structure, and the outer part is an outer shell. A notch part that is engaged with the guide strip is arranged in the middle of the outer shell. A pressing part is arranged at the bottom of the connecting part. When the pressing part is pressed upward, the bayonet pin is driven to extend and is inserted into the guide strip through the notch part. The connecting part is fixed by the friction between the bayonet pin and the guide strip. The pressing part and the outer shell are magnetically attracted to each other.

[0028] The pressing portion moves downward, driving the latch to withdraw from the notch portion, and at this time the connecting portion slides forward and backward relative to the guide bar.

[0029] In an optional embodiment, the data acquisition glove is provided with a plurality of inertial measurement units for collecting finger motion data and tactile sensors for collecting tactile perception data;

[0030] The inertial measurement units are all arranged on the back of the glove body, and the tactile sensors are all arranged on the front of the glove body.

[0031] In an optional embodiment, a plurality of wired posture sensors and at least one sensor interface are provided on the data acquisition motion capture suit; the signal line of the posture sensor is arranged in the lining of the suit and connected to the sensor interface; the host computer reads the data of the posture sensor by connecting the sensor interface via a data line.

[0032] In an optional embodiment, a pressure sensor is provided in each data collection shoe, the pressure sensor is integrated in the insole and placed in the shoe; the posture sensor is installed on the upper; after wearing the data collection shoe, the posture sensor collects posture information, and the pressure sensor records the sole pressure information;

[0033] The data acquisition shoes have a reserved data line connected to the host computer for sending the collected data to the host computer.

[0034] Compared with the prior art, the present invention has the following beneficial technical effects:

[0035] 1) High flexibility of data content: The data acquisition system of the present invention is a wearable data acquisition system that can be worn by a real operator to perform embodied data acquisition, and can obtain embodied intelligent data with flexible operation and high success rate.

[0036] 2) The data acquisition equipment is lightweight and has low deployment cost: The data acquisition system of the present invention is lightweight and can be easily worn, disassembled, and reassembled at any time. It can be easily deployed in multiple scenarios and collect embodied intelligent data in different scenarios at the same time.

[0037] 3) Diverse sensors and multimodal data: The data acquisition system of the present invention is deployed with multiple different types of sensors, which can collect multimodal embodied intelligent data including color visual images, depth visual images, fisheye lens images, sound and voice information, human joint postures, hand pressure information, foot pressure information, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present specification. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0039] Figure 1 A system structure diagram in an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of the overall structure of a data collection helmet in an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of the back structure of a data collection helmet according to an embodiment of the present invention;

[0042] Figure 4 Schematic diagram of the unfolded structure of the helmet body in an embodiment of the present invention;

[0043] Figure 5 Schematic diagram of the connection structure of the sliding assembly and the second bracket in an embodiment of the present invention;

[0044] Figure 6 This is one of the structural schematic diagrams of the connecting portion in an embodiment of the present invention;

[0045] Figure 7 This is the second structural schematic diagram of the connecting portion in an embodiment of the present invention.

[0046] Figure numerals: 100, data collection helmet; 200, data collection gloves; 300, data collection motion capture suit; 400, data collection shoes; 1, helmet body; 11, hard protective layer; 111, slot; 12, lining layer; 121, opening; 2, first bracket; 3, counterweight assembly; 4, sliding assembly; 41, base; 42, guide strip; 43, connecting part; 431, outer shell; 432, pressing part; 433, notch; 434, pin; 5, second bracket; 51, universal joint; 52, connecting rod; 6, ultra-wide-angle camera; 7, strap; 8, RealSense RGBD camera. DETAILED DESCRIPTION

[0047] In order to better understand the above-mentioned technical scheme, the technical scheme of the embodiments of the present application is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical scheme of the embodiments of the present application, rather than limitations on the technical scheme of the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments may be combined with each other.

[0048] Example 1

[0049] like Figure 1 As shown, a system for collecting multimodal embodied data proposed by the present invention includes a data collection helmet 100, a data collection glove 200, a data collection motion capture suit 300 and a data collection shoe 400;

[0050] The data collection helmet 100 is used to collect color visual images, depth visual images, sound voice information and / or fisheye lens images;

[0051] Two data collection gloves 200 are provided to collect the motion state of finger joints and the contact information between fingers and objects;

[0052] The data acquisition motion capture suit 300 is used to collect the whole body posture information of the human body;

[0053] The data collection shoe 400 is used to collect the position information and pressure information of the sole of the foot;

[0054] The user puts on the data collection helmet 100, the data collection gloves 200, the data collection motion capture suit 300, and the data collection shoes 400 in sequence to collect data information; the collected data information is uploaded to the host computer through a wired connection.

[0055] In this embodiment, the data acquisition glove 200 is provided with a plurality of inertial measurement units for collecting finger motion data and tactile sensors for collecting tactile perception data; the inertial measurement units are all arranged on the back of the glove body, and the tactile sensors are all arranged on the front of the glove body.

[0056] The data acquisition motion capture suit 300 is provided with several wired posture sensors and at least one sensor interface; the signal line of the posture sensor is arranged in the lining of the clothing and connected to the sensor interface; the host computer reads the data of the posture sensor through the data line connected to the sensor interface.

[0057] The data collection shoes 400 are provided with pressure sensors, which are integrated into the insoles and placed inside the shoes; the posture sensors are installed on the uppers; after wearing the data collection shoes 400, the posture sensors collect posture information, and the pressure sensors record the sole pressure information;

[0058] The data collection shoe 400 has a reserved data line connected to a host computer for sending the collected data to the host computer.

[0059] Example 2

[0060] like Figure 2-3 As shown, a system for collecting multi-modal embodied data proposed by the present invention includes a helmet body 1, a first bracket 2, a second bracket 5, a sliding component 4 and a counterweight component 3;

[0061] A first bracket 2 is provided and located at the front of the helmet body 1; a RealSense RGBD camera 8 is installed on the first bracket 2 to capture image information of the front of the helmet body 1; RealSense is a real-sensing technology, and the image information collected by the camera is 3D image information with depth of field.

[0062] The sliding assembly 4 is symmetrically arranged on the left and right sides of the helmet body 1; the second bracket 5 is installed on the sliding assembly 4 and slides relatively along the sliding assembly 4; Figure 5 As shown, the sliding assembly 4 includes a base 41, a guide bar 42 and a connecting portion 43; the base 41 is formed integrally with the hard protective layer 11 as a substrate; the guide bar 42 is arranged on the base 41, and the connecting portion 43 is arranged in pairs and both are installed on the guide bar 42 and slide relative to the guide bar 42; Figure 6 As shown, the connecting portion 43 is a cylindrical structure, with an outer shell 431 on the outside, and a notch 433 is provided in the middle of the outer shell 431 to engage with the guide strip 42; a pressing portion 432 is provided at the bottom of the connecting portion 43, and when the pressing portion 432 is pressed upward, the bayonet 434 is driven to extend and insert into the guide strip 42 through the notch 433, and the connecting portion 43 is fixed by the friction between the bayonet 434 and the guide strip 42; the pressing portion 432 and the outer shell 431 maintain magnetic attraction; Figure 7 As shown, the pressing portion 432 moves downward, driving the latch 434 to withdraw from the notch portion 433, and at this time, the connecting portion 43 slides forward and backward relative to the guide strip 42;

[0063] The second bracket 5 can be moved forward and backward to a suitable position through the sliding component 4, so as to facilitate the collection of different information.

[0064] In this embodiment, the counterweight assembly 3 is fixed at the rear end of the helmet body 1, and is used to level the center of gravity of the entire helmet body 1 so that it is located at the geometric center of the helmet body 1; at the same time, the counterweight assembly 3 is integrated with a power supply module for powering the sampling equipment; the power supply module is electrically connected to each electrical device through wires to ensure the normal operation of each sampling device.

[0065] In this embodiment, a matching strap 7 for fixing is provided on the helmet body 1; the tightness of the strap 7 is adjustable, and when in use, after the user puts on the helmet body 1, he can pull the strap 7 to tighten the helmet body 1.

[0066] Example 3

[0067] The system for collecting multimodal embodied data proposed by the present invention is different from that of embodiment 1 in that: Figure 4 As shown, the helmet body 1 includes an external hard protective layer 11 and an internal lining layer 12;

[0068] The first bracket 2, the sliding assembly 4 and the counterweight assembly 3 are all fixed on the hard protective layer 11; the inner lining layer 12 is a flexible cushion layer in direct contact with the head. The hard protective layer 11 is provided with a plurality of strip-shaped slots 111; the inner lining layer 12 is provided with a plurality of openings 121 for ventilation; the positions of the openings 121 correspond to the positions of the slots 111. The hard protective layer 11 provides a foundation for the installation of various devices, and it has a certain strength that is lower than the external impact and is not easily damaged; while the inner lining layer 12 is a flexible cushion layer, which is in direct contact with the head and is softer and more comfortable; at the same time, the hard protective layer 11 and the inner lining layer 12 are respectively provided with strip-shaped slots 111 and openings 121 for ventilation, which facilitates ventilation and sweating during use for users, making it more comfortable.

[0069] Example 4

[0070] The present invention proposes a system for collecting multimodal embodied data. Different from the embodiment 1, in the present embodiment, RGB cameras or ultra-wide-angle cameras 6 are respectively arranged on the second brackets 5 located on both sides. And the second bracket 5 is a serpentine bracket; in actual use, the RealSense RGBD camera 8 on the first bracket 2, and the first bracket 2 is a three-axis gimbal, plays an anti-shake role when collecting image information with 3D depth of field, and the picture is more stable; and the RGB camera or ultra-wide-angle camera 6 on the second bracket 5 can assist in collecting image information; the setting of the serpentine bracket allows the user to manually adjust the direction and angle of the second bracket 5, collect image information from different angles, and improve the collection range of image data.

[0071] Example 5

[0072] The present invention proposes a system for collecting multimodal embodied data. Different from the first embodiment, in the present embodiment, ultra-wide-angle cameras 6 or microphone arrays with integrated IMU posture sensors are respectively provided on the second brackets 5 located on both sides. The second bracket 5 includes a spherical universal joint 51, a connecting rod 52 and a fixture; the universal joint 51 is a universal joint with a damper, and the angle is kept fixed after the position is manually adjusted; one end of the connecting rod 52 is fixed on the universal joint 51, and the other end is provided with a fixture for fixed connection with the sampling device;

[0073] During actual use, the RealSense RGBD camera 8 on the first bracket 2, which is a three-axis gimbal, plays an anti-shake role when collecting image information with 3D depth of field, and the picture is more stable; the ultra-wide-angle camera 6 on the second bracket 5 can assist in collecting image information; the microphone array integrated with the IMU posture sensor can synchronously collect posture information and sound information, thereby realizing multi-modal data collection.

[0074] In the description of the present application, it is necessary to understand that the directions or positional relationships indicated by the terms "up", "down", "left", "right", "front", "back", etc. are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or unit referred to must have a specific direction, be constructed and operated in a specific orientation, and therefore, cannot be understood as a limitation on the present invention.

[0075] In the description of this specification, the description of the terms "one embodiment", "some embodiments", "specific embodiments", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0076] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A system for collecting multimodal embodied data, characterized in that: It comprises a data collection helmet (100), data collection gloves (200), a data collection motion capture suit (300) and data collection shoes (400); The data collection helmet (100) is used to collect color visual images, depth visual images, sound and voice information and / or fisheye lens images; Two data collection gloves (200) are provided, which are used to collect the movement state of finger joints and the contact information between fingers and objects; The data acquisition motion capture suit (300) is used to collect the whole body posture information of the human body; The data collection shoe (400) is used to collect the position information and pressure information of the sole of the foot; The user puts on the data collection helmet (100), the data collection gloves (200), the data collection motion capture suit (300), and the data collection shoes (400) in sequence and then collects data information; the collected data information is uploaded to the host computer through a wired connection.

2. A system for collecting multimodal embodied data according to claim 1, characterized in that: The data acquisition helmet (100) comprises a helmet body (1), a first bracket (2), a second bracket (5), a sliding assembly (4) and a counterweight assembly (3); A first bracket (2) is provided and is located at the front of the helmet body (1); a RealSense RGBD camera (8) is installed on the first bracket (2) to capture image information of the front of the helmet body (1); the first bracket (2) is a three-axis gimbal; The sliding components (4) are symmetrically arranged on the left and right sides of the helmet body (1); the second bracket (5) is installed on the sliding component (4) and slides relatively along the sliding component (4); and the second brackets (5) located on the two sides are respectively provided with RGB cameras or ultra-wide-angle cameras (6); The counterweight assembly (3) is fixed to the rear end of the helmet body (1) and is used to level the center of gravity of the entire helmet body (1) so that it is located at the geometric center of the helmet body (1); a power supply module is integrated in the counterweight assembly (3) and is used to supply power to the sampling device; the power supply module is electrically connected to various electrical devices via wires; A matching strap (7) for fixing is arranged on the helmet body (1).

3. A system for collecting multimodal embodied data according to claim 2, characterized in that: The helmet body (1) comprises an external hard protective layer (11) and an internal lining layer (12); The first bracket (2), the sliding assembly (4) and the counterweight assembly (3) are all fixed on the hard protective layer (11); The inner lining layer (12) is a flexible cushion layer in direct contact with the head; the hard protective layer (11) is provided with a plurality of strip-shaped slots (111); The inner lining layer (12) is provided with a plurality of openings (121) for ventilation; the positions of the openings (121) correspond to the positions of the slots (111).

4. A system for collecting multimodal embodied data according to claim 2, characterized in that: An ultra-wide-angle camera (6) or a microphone array integrated with an IMU posture sensor is respectively arranged on the second brackets (5) located on both sides.

5. A system for collecting multimodal embodied data according to claim 2, characterized in that: The second bracket (5) is a serpentine bracket.

6. A system for collecting multimodal embodied data according to claim 2, characterized in that: The second bracket (5) comprises a spherical universal joint (51), a connecting rod (52) and a fixer; the universal joint (51) is a universal joint with a damper, and the angle is kept fixed after the position is manually adjusted; One end of the connecting rod (52) is fixed on the universal joint (51), and the other end is provided with a fixer for fixedly connecting with the sampling device.

7. A system for collecting multimodal embodied data according to claim 2, characterized in that: The sliding assembly (4) comprises a base (41), a guide bar (42) and a connecting portion (43); The base (41) is formed integrally with the hard protective layer (11) as a base material; the guide bar (42) is arranged on the base (41); the connecting parts (43) are arranged in pairs and are both installed on the guide bar (42) and slide relative to the guide bar (42); The connecting part (43) is a cylindrical structure, and the outer part is an outer shell (431). A notch (433) that is engaged with the guide bar (42) is arranged in the middle of the outer shell (431); a pressing part (432) is arranged at the bottom of the connecting part (43); when the pressing part (432) is pressed upward, the latch (434) is driven to extend and is inserted into the guide bar (42) through the notch (433), and the connecting part (43) is fixed by the friction force between the latch (434) and the guide bar (42); the pressing part (432) and the outer shell (431) are magnetically attracted to each other; When the pressing portion (432) moves downward, the latch (434) is driven to withdraw from the notch portion (433), and at this time, the connecting portion (43) slides forward and backward relative to the guide bar (42).

8. The system for collecting multimodal embodied data according to claim 1, characterized in that: The data collection glove (200) is provided with a plurality of inertial measurement units for collecting finger motion data and tactile sensors for collecting tactile perception data; The inertial measurement units are all arranged on the back of the glove body, and the tactile sensors are all arranged on the front of the glove body.

9. A system for collecting multimodal embodied data according to claim 1, characterized in that: A plurality of wired posture sensors and at least one sensor interface are arranged on a data acquisition motion capture suit (300); signal lines of the posture sensors are arranged in the lining of the suit and connected to the sensor interface; a host computer reads data from the posture sensors by connecting the sensor interface via a data line.

10. A system for collecting multimodal embodied data according to claim 1, characterized in that: The data collection shoes (400) are provided with pressure sensors, which are integrated into the insoles and placed inside the shoes; the posture sensors are installed on the uppers; after wearing the data collection shoes (400), the posture sensors collect posture information, and the pressure sensors record the sole pressure information; The data collection shoe (400) has a reserved data line for connection with a host computer, and is used to send the collected data to the host computer.

Images