Data acquisition system, terminal and method for end effector of humanoid robot

Abstract

The embodiment of the invention provides a data acquisition system, terminal and method for an end effector of a humanoid robot. The system comprises at least one data acquisition terminal and a data collection device, wherein the data collection device is respectively connected with each data acquisition terminal; the data acquisition terminal is provided with an actuator corresponding to an end actuator of the humanoid robot; the data acquisition terminal is used for acquiring process data corresponding to a scene task in the process of executing the scene task, and sending the process data to the data collection device; the data collection device is used for collecting process data to obtain scene task data corresponding to at least one scene task, the scene task data corresponding to each scene task is used for training an artificial intelligence (AI) model, and the AI model obtained by training is used for controlling the humanoid robot to execute the scene task. The system can complete data acquisition without depending on a robot body, and is high in adaptability.

Description

Technical Field

[0001] The present application relates to robotics technology, and in particular to a data acquisition system, terminal and method for an end effector of a humanoid robot. Background Art

[0002] Embodied data plays a key role in the training of robot embodied intelligence models and is the core foundation for realizing robot embodied intelligence. At present, the commonly used robot embodied data collection methods mainly rely on teleoperation technology. Common teleoperation methods include teleoperation based on virtual reality (VR), teleoperation based on motion capture technology, and teleoperation based on drag-and-drop teaching. These methods can effectively capture the robot's motion characteristics and behavior patterns, providing rich data support for the training of embodied intelligence models.

[0003] Teleoperation is strictly corresponding to the robot body. Different types of robots rely on specific teleoperation methods to be controlled by human operators and complete corresponding tasks through remote operation. However, the embodied data collected through teleoperation is only applicable to specific robot bodies, lacks the ability to generalize across robot platforms, and has poor adaptability. Summary of the invention

[0004] The embodiments of the present application provide a data acquisition system, a terminal, and a method for an end effector of a humanoid robot, so as to achieve a highly adaptable effect.

[0005] In a first aspect, an embodiment of the present application provides a data acquisition system for an end effector of a humanoid robot, comprising:

[0006] At least one data acquisition terminal and data collection device;

[0007] Wherein, the data collection device is connected to each of the data collection terminals respectively; the data collection terminal has an actuator corresponding to the end effector of the humanoid robot;

[0008] The data acquisition terminal is used to collect process data corresponding to the scenario task during the execution of the scenario task, and send the process data to the data collection device;

[0009] The data collection device is used to collect the process data collected by the data collection terminal to obtain scene task data corresponding to at least one scene task; the scene task data corresponding to each of the scene tasks is used to train an artificial intelligence AI model, and the trained AI model is used to control the humanoid robot to perform scene tasks.

[0010] In a possible implementation, the system further includes:

[0011] A training device connected to the data collection device, wherein the training device is used to train the AI ​​model according to the scenario task data corresponding to each of the scenario tasks.

[0012] In a possible implementation manner, the data collection device is further used for:

[0013] Determine a data collection mode and a data collection terminal corresponding to the data collection mode;

[0014] Sending first indication information to a data acquisition terminal corresponding to the data acquisition mode, wherein the first indication information is used to instruct the data acquisition terminal to perform an initialization process;

[0015] The data acquisition terminal is further used to send response information corresponding to the first indication information to the data collection device after completing the initialization process.

[0016] In a possible implementation, the data collection device is further used to send second indication information to the data collection terminal when the response information is a successful response information, and the second indication information is used to instruct the data collection terminal to start collecting data.

[0017] In a possible implementation, the process data includes at least one of the following: location information of the data acquisition terminal, posture information of the data acquisition terminal, opening and closing state information of an actuator of the data acquisition terminal, and scene image data.

[0018] In a possible implementation manner, the data acquisition terminal is specifically used for:

[0019] Determining initial position information and initial posture information of the data acquisition terminal according to a plurality of scene images acquired by the data acquisition terminal;

[0020] Obtaining the position information and attitude information of the origin of the reference coordinate system according to the position information and attitude information of the preset position in the scene where the data acquisition terminal is located;

[0021] Establishing a mapping relationship between the initial position information and the position information of the origin, and between the initial posture information and the posture information of the origin;

[0022] According to the mapping relationship, the initial position information and the initial posture information are respectively converted into the position information and the posture information in the reference coordinate system.

[0023] In a possible implementation manner, the data acquisition terminal is specifically used for:

[0024] Extracting key feature points from each of the scene images, and matching the key feature points in each of the scene images to obtain the spatial position of each of the key feature points in different scene images;

[0025] The initial position information and initial posture information of the data acquisition terminal are determined according to the spatial position of each of the key feature points in the different scene images.

[0026] In a second aspect, an embodiment of the present application provides a data collection terminal, including:

[0027] Actuator, opening and closing control component, image acquisition component and processing component;

[0028] Wherein, the opening and closing control component is connected to the actuator, and the opening and closing control component is used to control the opening and closing degree of the actuator;

[0029] The actuator is used to perform corresponding actions through the control of the opening and closing control component;

[0030] The image acquisition component is connected to the processing component, and the image acquisition component is used to acquire scene image data of the data acquisition terminal during the execution of the scene task;

[0031] The processing component is used to obtain process data corresponding to the scene task, and the process data includes at least one of the following: position information of the data acquisition terminal, posture information of the data acquisition terminal, opening and closing state information of the actuator and the scene image data.

[0032] In a possible implementation manner, the processing component is further configured to:

[0033] Determining initial position information and initial posture information of the image acquisition component according to a plurality of scene images acquired by the image acquisition component;

[0034] Obtaining the position information and attitude information of the origin of the reference coordinate system according to the position information and attitude information of the preset position in the scene where the data acquisition terminal is located;

[0035] Establishing a mapping relationship between the initial position information and the position information of the origin, and between the initial posture information and the posture information of the origin;

[0036] According to the mapping relationship, the initial position information and the initial posture information are respectively converted into the position information and the posture information in the reference coordinate system.

[0037] In a possible implementation manner, the processing component is specifically used for:

[0038] Extracting key feature points from each of the scene images, and matching the key feature points in each of the scene images to obtain the spatial position of each of the key feature points in different scene images;

[0039] According to the spatial position of each of the key feature points in the different scene images, the initial position information and initial posture information of the image acquisition component are determined.

[0040] In a possible implementation manner, the processing component is further configured to:

[0041] The initial position information and initial posture information of the actuator are determined according to the position and posture mapping relationship between the image acquisition component and the actuator and the initial position information and initial posture information of the image acquisition component.

[0042] In a possible implementation manner, the structure of the actuator adopts at least one of the following: a two-finger gripper structure, a three-finger gripper structure, or a dexterous hand structure.

[0043] In a third aspect, an embodiment of the present application provides a data collection method for an end effector of a humanoid robot, which is applied to the system in the first aspect and / or various possible implementations of the first aspect, and the method includes:

[0044] The data acquisition terminal collects process data corresponding to the scenario task during the execution of the scenario task, and sends the process data to the data collection device;

[0045] The data collection device collects the process data collected by the data collection terminal to obtain scene task data corresponding to at least one scene task; the scene task data corresponding to each of the scene tasks is used to train an artificial intelligence AI model, and the trained AI model is used to control the humanoid robot to perform scene tasks.

[0046] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to implement the method of the third aspect above when executed by a processor.

[0047] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements the method of the third aspect above when executed by a processor.

[0048] The embodiment of the present application provides a data acquisition system, terminal and method for the end effector of a humanoid robot, the system comprising: at least one data acquisition terminal and a data collection device; wherein the data collection device is respectively connected to each data acquisition terminal; the data acquisition terminal is used to collect process data corresponding to a scene task during the execution of a scene task, and send the process data to the data collection device; since the data acquisition terminal has an actuator corresponding to the end effector of the humanoid robot, the data collected by the data acquisition terminal does not depend on the humanoid robot body, and the data acquisition terminal matches the humanoid robot with the corresponding end effector, therefore, the collected data can meet the training requirements of the humanoid robot model, and has strong adaptability, further, the data collection device is used to collect the process data collected by the data acquisition terminal, and obtain scene task data corresponding to at least one scene task; the scene task data corresponding to each of the scene tasks is used to train an artificial intelligence AI model, and the trained AI model is used to control the humanoid robot to perform scene tasks, therefore, the above scheme provides rich data support for the training of the robot AI model, and the data acquisition efficiency is high. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0050] Figure 1 A schematic diagram of the structure of the data acquisition system of the end effector of the humanoid robot provided in this application;

[0051] Figure 2 A schematic diagram of the implementation principle of the data acquisition system of the end effector of the humanoid robot provided in this application;

[0052] Figure 3 Schematic diagram of the process of the data acquisition system of the end effector of the humanoid robot provided in this application Figure 1 ;

[0053] Figure 4 Schematic diagram of the process of the data acquisition system of the end effector of the humanoid robot provided in this application Figure 2 ;

[0054] Figure 5 Schematic diagram of the process of the data acquisition system of the end effector of the humanoid robot provided in this application Figure 3 ;

[0055] Figure 6 The structure diagram of the data acquisition terminal provided for this application Figure 1 ;

[0056] Figure 7The structure diagram of the data acquisition terminal provided for this application Figure 2 ;

[0057] Figure 8 Schematic diagram of the data collection method provided for this application Figure 1 .

[0058] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0059] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0060] First, the application scenarios involved in this application are introduced:

[0061] The specific application scenario of this application is to collect process data of the end effector when performing scene tasks for humanoid robots, pan-humanoid robots or embodied robots, including: position information, posture information, opening and closing state information of the end effector or scene image data, etc.

[0062] Combining the above scenarios, it can be seen that in the existing technology, the collected embodied data is only applicable to a specific robot body, lacks the generalization ability across robot platforms, and has poor adaptability.

[0063] The data acquisition system provided in the present application collects data during the execution of scene tasks through a data acquisition terminal having an actuator corresponding to the end effector of the humanoid robot. It can adapt to different robot bodies and has strong adaptability.

[0064] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0065] Figure 1 The schematic diagram of the structure of the data acquisition system of the end effector of the humanoid robot provided in this application is as follows: Figure 1 As shown, the system includes:

[0066] At least one data collection terminal 100 and a data collection device 200;

[0067] Wherein, the data collection device is connected to each of the data collection terminals respectively; the data collection terminal has an actuator corresponding to the end effector of the humanoid robot;

[0068] The data acquisition terminal is used to collect process data corresponding to the scenario task during the execution of the scenario task, and send the process data to the data collection device;

[0069] The data collection device is used to collect the process data collected by the data collection terminal to obtain scene task data corresponding to at least one scene task; the scene task data corresponding to each of the scene tasks is used to train an artificial intelligence AI model, and the trained AI model is used to control the humanoid robot to perform scene tasks.

[0070] Specifically, the data acquisition terminal has an actuator corresponding to the end effector of the humanoid robot. For example, the actuator of the data acquisition terminal needs to be consistent with the end effector of the humanoid robot in opening and closing mode and similar in shape. For example, if the humanoid robot is a two-finger gripper, then the actuator of the data acquisition terminal also adopts a similar two-finger gripper structure, for example, a lightweight gripper structure can be constructed by 3D printing.

[0071] Optionally, the structure of the actuator of the data acquisition terminal adopts at least one of the following: a two-finger gripper structure, a three-finger gripper structure or a dexterous hand structure.

[0072] During the execution of the scenario task, the data acquisition terminal acquires process data corresponding to the scenario task and sends the acquired process data to the data collection device.

[0073] Optionally, the process data includes at least one of the following: location information of the data acquisition terminal, posture information of the data acquisition terminal, opening and closing state information of an actuator of the data acquisition terminal, and scene image data.

[0074] Optionally, the location information of the data collection terminal may be represented as a location trajectory.

[0075] The opening and closing state information of the actuator includes, for example, the opening and closing state (eg, open or closed state), the opening and closing degree and other information.

[0076] Optionally, the process data has corresponding time stamp information.

[0077] Optionally, the data acquisition terminal sends the collected process data to the data collection device via a wired or wireless connection.

[0078] Furthermore, the data collection device integrates the collected process data to obtain scene task data corresponding to different scene tasks. The scene task data corresponding to each scene task is used to further train the AI ​​model. The trained AI model will be able to drive the robot to complete the corresponding scene tasks.

[0079] Optionally, the data acquisition terminal can be a handheld terminal to facilitate the completion of scene tasks and data collection.

[0080] The data acquisition system provided in the embodiment of the present application includes: at least one data acquisition terminal and a data collection device; wherein the data collection device is connected to each data acquisition terminal respectively; the data acquisition terminal is used to collect process data corresponding to the scene task during the execution of the scene task, and send the process data to the data collection device; since the data acquisition terminal has an actuator corresponding to the end effector of the humanoid robot, the data collected by the data acquisition terminal does not depend on the humanoid robot body, and the data acquisition terminal matches the humanoid robot with the corresponding end effector, therefore, the collected data can meet the training requirements of the humanoid robot model and has strong adaptability. Furthermore, the data collection device is used to collect the process data collected by the data acquisition terminal to obtain scene task data corresponding to at least one scene task; the scene task data corresponding to each of the scene tasks is used to train the artificial intelligence AI model, and the trained AI model is used to control the humanoid robot to perform the scene task. Therefore, the above scheme provides rich data support for the training of the robot AI model and has high data acquisition efficiency.

[0081] In some embodiments, Figure 2 As shown, the system also includes:

[0082] A training device connected to the data collection device, wherein the training device is used to train the AI ​​model according to the scenario task data corresponding to each of the scenario tasks.

[0083] Specifically, the data collection device sends the scene task data corresponding to each of the scene tasks to the training device; the training device trains the artificial intelligence AI model according to the scene task data corresponding to each of the scene tasks, and the trained AI model is used to control the humanoid robot to perform the scene tasks.

[0084] In the above embodiment, the training device trains the AI ​​model based on the scene task data corresponding to each scene task collected by the data collection device, so that the trained AI model can control the humanoid robot to perform the corresponding scene tasks, thereby improving the reliability and accuracy of the humanoid robot in performing tasks.

[0085] In some embodiments, the data collection device is further used to:

[0086] Determine a data collection mode and a data collection terminal corresponding to the data collection mode;

[0087] Sending first indication information to a data acquisition terminal corresponding to the data acquisition mode, wherein the first indication information is used to instruct the data acquisition terminal to perform an initialization process;

[0088] The data acquisition terminal is further used to send response information corresponding to the first indication information to the data collection device after completing the initialization process.

[0089] Specifically, in response to the user's instructions, the data collection mode is determined, for example, the data collection mode includes at least one of the following: single-arm mode, single-arm plus top camera mode, dual-arm collaborative mode, dual-arm collaborative with top camera mode; wherein, the single-arm mode refers to the use of one data collection terminal, the single-arm plus top camera mode refers to a data collection terminal plus a head-mounted camera, the dual-arm collaborative mode refers to two data collection terminals cooperating to collect data, and the dual-arm collaborative with top camera mode refers to the cooperation of two data collection terminals plus a head-mounted camera.

[0090] After determining the data acquisition mode, determine the corresponding data acquisition terminal, and send the first indication information to the corresponding data acquisition terminal. The first indication information is used to instruct the data acquisition terminal to perform an initialization process, such as initializing the reference coordinate system, converting the position and posture determined by the data acquisition terminal to the position and posture in the reference coordinate system, etc.

[0091] After completing the initialization process, the data acquisition terminal sends response information corresponding to the first indication information to the data collection device, for example, indicating whether the initialization process is completed.

[0092] In some embodiments, the data collection device is further used to send second indication information to the data collection terminal when the response information is a successful response information, and the second indication information is used to instruct the data collection terminal to start collecting data.

[0093] Specifically, when receiving the successful response information sent by the data collection terminal, the data collection device sends the second instruction information to the data collection terminal to instruct the data collection terminal to start collecting data, that is, the data collection terminal executes the scene task and collects data.

[0094] In the above implementation, the data acquisition terminal and the data collection device interact with each other to complete the initialization process before data acquisition, which has low implementation complexity and high efficiency.

[0095] In some embodiments, the initialization process is as follows:

[0096] Determining initial position information and initial posture information of the data acquisition terminal according to a plurality of scene images acquired by the data acquisition terminal;

[0097] Obtaining the position information and attitude information of the origin of the reference coordinate system according to the position information and attitude information of the preset position in the scene where the data acquisition terminal is located;

[0098] Establishing a mapping relationship between the initial position information and the position information of the origin, and between the initial posture information and the posture information of the origin;

[0099] According to the mapping relationship, the initial position information and the initial posture information are respectively converted into the position information and the posture information in the reference coordinate system.

[0100] Specifically, a plurality of scene images are acquired, and initial position information and initial posture information of the data acquisition terminal are determined according to the plurality of scene images, that is, initial position information and initial posture information in a camera coordinate system;

[0101] The position information and posture information of a preset position in the scene where the data acquisition terminal is located are obtained, and the position information and posture information of the preset position are used as the position information and posture information of the origin of the reference coordinate system.

[0102] Establish the mapping relationship between the initial position information of the data acquisition terminal and the position information of the origin, as well as the initial posture information and the posture information of the origin, and according to the mapping relationship, convert the initial position information of the data acquisition terminal into the position information in the reference coordinate system, and convert the initial posture information of the data acquisition terminal into the posture information in the reference coordinate system.

[0103] Optionally, the data acquisition terminal continuously acquires scene images while executing scene tasks, and determines the real-time position information and posture information of the data acquisition terminal based on the scene images, and based on the above mapping relationship, it can be converted into real-time position information and posture information in the reference coordinate system.

[0104] In the above implementation, by converting the position information and posture information of the data acquisition terminal into the position information and posture information in the reference coordinate system, it is better matched with the scene and the collected data is more in line with actual needs.

[0105] For example, Figure 3 As shown, the single-arm mode data acquisition workflow is as follows:

[0106] The single-arm mode requires the use of a handheld data collection terminal and its connection to the data collection device. A positioning QR code is posted in the data collection scene. The positioning QR code contains the location information and posture information of the current location to ensure the continuity of the data collection terminal position.

[0107] 1. Select the mode:

[0108] Select "Single-arm mode" in the data collection device and specify the corresponding data collection terminal. The actuator of the data collection terminal needs to be consistent with the opening and closing method of the corresponding robot end effector and have a similar shape. For example, if the corresponding robot is a two-finger gripper, then the actuator of the data collection terminal also needs to be a similar two-finger gripper structure.

[0109] 2. Initialize the coordinate system:

[0110] Clicking the "Ready" button in the data collection device sends the first indication information to the data collection terminal. After receiving the first indication information, the data collection terminal records the position information and posture information of the current actuator (such as the gripper) and records them as the initial position and initial posture. The user (or operator) scans the positioning QR code through the handheld data collection terminal to perform the coordinate system initialization process. After scanning the positioning QR code, the position and posture of the positioning QR code are determined as the position and posture of the origin of the reference coordinate system for data recording, and the reference coordinate system for data recording is established, and the mapping relationship between the position and posture of the origin and the initial position and initial posture of the actuator is calculated. After completing the initialization of the reference coordinate system for data recording, the data collection terminal sends a response message to the data collection device to indicate whether the initialization process is completed. The data collection device can issue a second indication message (i.e., a start instruction) to instruct the data collection terminal to start collecting data.

[0111] 3. Execute the task:

[0112] After the data collection device sends the second instruction information (i.e., the start instruction), the operator completes the corresponding scene task through the data collection terminal.

[0113] During the task execution, the data acquisition terminal continuously records the position information, posture information, the opening and closing status information of the end effector, the scene image data captured by the camera, and the corresponding timestamp information.

[0114] The data is sent via a wired or wireless connection to a data collection device, which continuously records the process data with a time stamp.

[0115] 4. End data recording:

[0116] After completing the scenario task, click the "Finish" button on the handheld data acquisition terminal to send an end instruction to the data collection device. The data collection device stops recording and packages all collected data into scenario task data corresponding to a scenario task, and sends the scenario task data to the training device.

[0117] 5. Model training:

[0118] When scenario task data corresponding to more than a preset number of scenario tasks are received, these scenario task data are input into the training device to further train the AI ​​model. The trained model will be able to drive the robot to complete the corresponding scenario tasks.

[0119] For example, Figure 2 , Figure 4 As shown, the data collection workflow of the dual-arm collaborative top camera mode is as follows:

[0120] This mode requires the use of two handheld data collection terminals and a head-mounted camera, which are connected to the data collection device. Positioning QR codes are posted in the data collection scene to ensure the continuity of the data collection terminal location.

[0121] 1. Select the mode:

[0122] Select "Dual-arm collaboration with top camera mode" in the data collection device, specify the corresponding two data collection terminals and connect the cameras to the data collection terminals and the data collection device. The actuator of the data collection terminal needs to be consistent with the opening and closing method and shape of the corresponding robot end effector.

[0123] 2. Initialize the coordinate system:

[0124] Clicking the "Ready" button in the data collection device sends the first indication information to the two data collection terminals. After receiving the first indication information, the data collection terminal records the position information and posture information of the current actuator (such as the gripper) and records them as the initial position and initial posture. The user scans the positioning QR code through the handheld data collection terminal to perform the coordinate system initialization process. After scanning the positioning QR code, the position and posture of the positioning QR code are determined as the position and posture of the origin of the reference coordinate system for data recording, and the reference coordinate system for data recording is established. After completing the initialization of the reference coordinate system for data recording and ensuring that the head-mounted camera has image data transmission, the data collection terminal sends a response message to the data collection device to indicate whether the initialization process is completed, that is, whether it is ready. The data collection device can send a second indication message (i.e., a start instruction) to instruct the data collection terminal to start collecting data.

[0125] 3. Execute the task:

[0126] After the data collection device sends the second instruction information (i.e., the start instruction), the operator completes the corresponding scene task through the data collection terminal, and at the same time requires the head-mounted camera to capture the operation process of at least one handheld data collection terminal, that is, at least one handheld data collection terminal needs to remain in the picture of the head-mounted camera.

[0127] During the execution of the task, the data acquisition terminal continuously records and transmits process data, including at least one of the following: position information, posture information, opening and closing state information of the end effector, scene image data captured by the camera of the data acquisition terminal, and corresponding timestamp information. The head-mounted camera continuously records and transmits scene image data with timestamps.

[0128] The process data is sent via a wired or wireless connection to a data collection device, which continuously records the process data with a time stamp.

[0129] 4. End data recording:

[0130] After completing the scenario task, click the "Finish" button on the handheld data acquisition terminal to send an end instruction to the data collection device. The data collection device stops recording and packages all collected data into scenario task data corresponding to a scenario task, and sends the scenario task data to the training device.

[0131] 5. Model training:

[0132] When scenario task data corresponding to more than a preset number of scenario tasks are received, these scenario task data are input into the training device to further train the AI ​​model. The trained model will be able to drive the robot to complete the corresponding scenario tasks.

[0133] In some embodiments, the data collection terminal is specifically used for:

[0134] Extracting key feature points from each of the scene images, and matching the key feature points in each of the scene images to obtain the spatial position of each of the key feature points in different scene images;

[0135] The initial position information and initial posture information of the data acquisition terminal are determined according to the spatial position of each of the key feature points in the different scene images.

[0136] Specifically, the scene image is acquired through the image acquisition component of the data acquisition terminal, for example, a plurality of continuous scene image frames are acquired.

[0137] Extract key feature points from multiple scene image frames, such as corner points, edges or textures; match the key feature points extracted from consecutive scene image frames to determine the spatial positions of the same key feature points in different scene image frames.

[0138] Based on the spatial positions of the matched key feature points in different scene image frames, a perspective-n-point (PNP) algorithm is used to estimate the position and posture of the current image acquisition component. Optionally, the position and posture of the image acquisition component are used as the position and posture of the data acquisition terminal.

[0139] Optionally, after the first position and attitude calculation is completed, the process is repeated continuously to update the real-time position and attitude of the data acquisition terminal.

[0140] For example, Figure 5 As shown, the initialization process is as follows:

[0141] S1, input continuous image frames;

[0142] S2, extracting key feature points in the image frame;

[0143] S3, matching the extracted feature points to determine the spatial positions of the same feature points in different image frames;

[0144] S4, use the PNP algorithm to estimate the current camera position and posture;

[0145] S5: Whether the positioning QR code is scanned;

[0146] S6, coordinate system mapping; that is, establishing a mapping relationship between the position and posture of the data acquisition terminal in the camera coordinate system and the position and posture of the origin in the reference coordinate system.

[0147] S7. Output the current position and attitude of the camera in the reference coordinate system.

[0148] Here, the camera refers to the image acquisition component in the data acquisition terminal.

[0149] Figure 6 Schematic diagram of the data acquisition terminal structure of the data acquisition system of the end effector of the humanoid robot provided in this application Figure 1 ,like Figure 6 As shown, the data acquisition terminal includes:

[0150] Actuator 10, opening and closing control component 11, image acquisition component 12 and processing component (not shown in the figure);

[0151] The opening and closing control component 11 is connected to the actuator 10, and the opening and closing control component 11 is used to control the opening and closing degree of the actuator 10;

[0152] The actuator 10 is used to perform corresponding actions through the control of the opening and closing control component 11;

[0153] The image acquisition component 12 is connected to the processing component, and the image acquisition component 12 is used to acquire scene image data of the data acquisition terminal during the execution of the scene task;

[0154] The processing component is used to obtain process data corresponding to the scene task, and the process data includes at least one of the following: position information of the data acquisition terminal, posture information of the data acquisition terminal, opening and closing state information of the actuator and the scene image data.

[0155] Specifically, Figure 6 , Figure 7 The structure of the data acquisition terminal shown in the figure is only an example, and the shapes and connections of the various components may also be in other ways, which are not limited in the embodiments of the present application.

[0156] The data acquisition terminal has an actuator corresponding to the end effector of the humanoid robot. For example, the actuator of the data acquisition terminal needs to be consistent with the end effector of the humanoid robot in opening and closing mode and similar in shape. For example, if the humanoid robot is a two-finger gripper, then the actuator of the data acquisition terminal also adopts a similar two-finger gripper structure.

[0157] Optionally, the structure of the actuator of the data acquisition terminal adopts at least one of the following: a two-finger gripper structure, a three-finger gripper structure or a dexterous hand structure.

[0158] Among them, the opening and closing control component is used to control the opening and closing action of the actuator, that is, to control the actuator to perform corresponding actions.

[0159] During the execution of the scene task, the data acquisition terminal controls the actuator to perform operations corresponding to the scene task through the opening and closing control component, the image acquisition component collects scene image data during the execution of the scene task, and the processing component records the process data during the execution of the scene task, for example, including at least one of the following: location information, posture information, opening and closing status information, scene image data, etc. of the data acquisition terminal.

[0160] Optionally, the process data includes at least one of the following: location information of the data acquisition terminal, posture information of the data acquisition terminal, opening and closing state information of an actuator of the data acquisition terminal, and scene image data.

[0161] Optionally, the location information of the data collection terminal may be represented as a location trajectory.

[0162] The opening and closing state information of the actuator includes, for example, opening and closing state (such as open or closed), opening and closing degree and other information.

[0163] Optionally, the process data has corresponding time stamp information.

[0164] Optionally, the data acquisition terminal further includes: a switch button for controlling the start or shut down of the data acquisition terminal.

[0165] Optionally, the data acquisition terminal further includes: a power supply component, used to supply power to the data acquisition terminal.

[0166] The implementation principle and technical effect of the data acquisition terminal provided in this embodiment are similar to those of the data acquisition system, and are not described in detail in this embodiment.

[0167] In some embodiments, the data collection terminal may implement the initialization process in the following manner, that is, the processing component is further used to:

[0168] Determining initial position information and initial posture information of the image acquisition component according to a plurality of scene images acquired by the image acquisition component;

[0169] Obtaining the position information and attitude information of the origin of the reference coordinate system according to the position information and attitude information of the preset position in the scene where the data acquisition terminal is located;

[0170] Establishing a mapping relationship between the initial position information and the position information of the origin, and between the initial posture information and the posture information of the origin;

[0171] According to the mapping relationship, the initial position information and the initial posture information are respectively converted into the position information and the posture information in the reference coordinate system.

[0172] In some embodiments, the processing component is specifically configured to:

[0173] Extracting key feature points from each of the scene images, and matching the key feature points in each of the scene images to obtain the spatial position of each of the key feature points in different scene images;

[0174] According to the spatial position of each of the key feature points in the different scene images, the initial position information and initial posture information of the image acquisition component are determined.

[0175] In some embodiments, the processing component is further configured to:

[0176] The initial position information and initial posture information of the actuator are determined according to the position and posture mapping relationship between the image acquisition component and the actuator and the initial position information and initial posture information of the image acquisition component.

[0177] Specifically, the actuator is fixedly connected to the image acquisition component (such as a camera), so the coordinate transformation matrix of the posture of the image acquisition component and the posture of the actuator can be obtained to establish a posture mapping relationship.

[0178] Further, according to the position-pose mapping relationship, and the initial position information and initial posture information of the image acquisition component, the initial position information and initial posture information of the actuator are determined.

[0179] Optionally, the process data may include: position information and posture information of the image acquisition component and / or the actuator. Alternatively, the position information and posture information of the data acquisition terminal may be obtained based on the position information and posture information of the image acquisition component and the actuator.

[0180] In summary, the embodiment of the present application uses a data acquisition terminal to collect the position information, posture information, opening and closing state information of the actuator and related image data of the robot's end effector, trains the task model through an AI algorithm, and controls the embodied intelligent body through the trained model to complete the scene task. Compared with the traditional embodied data acquisition scheme, data acquisition can be completed without relying on the robot body through the data acquisition terminal, which lowers the threshold for data acquisition. At the same time, since the handheld operation is more flexible, the diversity of embodied data is greatly improved. By replacing the actuator part of the data acquisition terminal, the replacement of end effectors such as two-finger grippers, three-finger grippers or dexterous hands can be completed, and it can adapt to a variety of robots with strong adaptability. Since the model is trained through scene task data, it can be solved through inverse kinematics for different robot bodies, and the same model can be adapted to different robot bodies, which has strong adaptability.

[0181] In some embodiments, the data acquisition terminal may further include: at least one processor and a memory, wherein the processor, the memory, and the image acquisition component are connected via a bus.

[0182] In a specific implementation process, at least one processor executes the computer execution instructions stored in the memory, so that the at least one processor executes the data collection process of the above-mentioned data collection terminal.

[0183] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor.

[0184] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0185] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0186] Figure 8 Schematic diagram of the data collection method provided for this application Figure 1 The method of this embodiment is applied to the system in any of the above embodiments, such as Figure 8 As shown, the method includes:

[0187] S801, when executing a scene task, the data collection terminal collects process data corresponding to the scene task and sends the process data to the data collection device;

[0188] S802. The data collection device collects the process data collected by the data collection terminal to obtain scene task data corresponding to at least one scene task; the scene task data corresponding to each scene task is used to train the artificial intelligence AI model, and the trained AI model is used to control the humanoid robot to perform the scene task.

[0189] The method provided in this embodiment can be applied to the above-mentioned system or terminal embodiments. Its implementation principle and technical effects are similar, and are not described in detail in this embodiment.

[0190] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0191] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0192] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0193] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0194] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0195] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0196] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0197] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0198] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0199] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A data acquisition system for an end effector of a humanoid robot, characterized in that: include: At least one data acquisition terminal and data collection device; Wherein, the data collection device is connected to each of the data collection terminals respectively; the data collection terminal has an actuator corresponding to the end effector of the humanoid robot; The data acquisition terminal is used to collect process data corresponding to the scenario task during the execution of the scenario task, and send the process data to the data collection device; The data collection device is used to collect the process data collected by the data collection terminal to obtain scene task data corresponding to at least one scene task; the scene task data corresponding to each of the scene tasks is used to train an artificial intelligence AI model, and the trained AI model is used to control the humanoid robot to perform scene tasks.

2. The system according to claim 1, characterized in that The system further comprises: A training device connected to the data collection device, the training device is used to train the AI ​​model according to the scene task data corresponding to each of the scene tasks; the process data includes at least one of the following: position information of the data acquisition terminal, posture information of the data acquisition terminal, opening and closing state information of the actuator of the data acquisition terminal and scene image data.

3. The system according to claim 1 or 2, characterized in that: The data collection device is further used for: Determine a data collection mode and a data collection terminal corresponding to the data collection mode; Sending first indication information to a data acquisition terminal corresponding to the data acquisition mode, wherein the first indication information is used to instruct the data acquisition terminal to perform an initialization process; The data acquisition terminal is further used to send response information corresponding to the first indication information to the data collection device after completing the initialization process.

4. The system according to claim 3, characterized in that The data collection device is further used to send second indication information to the data collection terminal when the response information is successful response information, and the second indication information is used to instruct the data collection terminal to start collecting data.

5. The system according to claim 3, characterized in that The data acquisition terminal is specifically used for: Determining initial position information and initial posture information of the data acquisition terminal according to a plurality of scene images acquired by the data acquisition terminal; Obtaining the position information and attitude information of the origin of the reference coordinate system according to the position information and attitude information of the preset position in the scene where the data acquisition terminal is located; Establishing a mapping relationship between the initial position information and the position information of the origin, and between the initial posture information and the posture information of the origin; According to the mapping relationship, the initial position information and the initial posture information are respectively converted into the position information and the posture information in the reference coordinate system.

6. The system according to claim 5, characterized in that The data acquisition terminal is specifically used for: Extracting key feature points from each of the scene images, and matching the key feature points in each of the scene images to obtain the spatial position of each of the key feature points in different scene images; The initial position information and initial posture information of the data acquisition terminal are determined according to the spatial position of each of the key feature points in the different scene images.

7. A data collection terminal, characterized in that: include: Actuator, opening and closing control component, image acquisition component and processing component; Wherein, the opening and closing control component is connected to the actuator, and the opening and closing control component is used to control the opening and closing degree of the actuator; The actuator is used to perform corresponding actions through the control of the opening and closing control component; The image acquisition component is connected to the processing component, and the image acquisition component is used to acquire scene image data of the data acquisition terminal during the execution of the scene task; The processing component is used to obtain process data corresponding to the scene task, and the process data includes at least one of the following: position information of the data acquisition terminal, posture information of the data acquisition terminal, opening and closing state information of the actuator and the scene image data.

8. The data acquisition terminal according to claim 7, characterized in that: The processing component is also used to: Determining initial position information and initial posture information of the image acquisition component according to a plurality of scene images acquired by the image acquisition component; Obtaining the position information and attitude information of the origin of the reference coordinate system according to the position information and attitude information of the preset position in the scene where the data acquisition terminal is located; Establishing a mapping relationship between the initial position information and the position information of the origin, and between the initial posture information and the posture information of the origin; According to the mapping relationship, the initial position information and the initial posture information are respectively converted into the position information and the posture information in the reference coordinate system.

9. The data acquisition terminal according to claim 8, characterized in that: The processing component is specifically used for: Extracting key feature points from each of the scene images, and matching the key feature points in each of the scene images to obtain the spatial position of each of the key feature points in different scene images; According to the spatial position of each of the key feature points in the different scene images, the initial position information and initial posture information of the image acquisition component are determined.

10. A data collection method for an end effector of a humanoid robot, characterized in that: Applied to the system according to any one of claims 1 to 6, the method comprises: The data acquisition terminal collects process data corresponding to the scenario task during the execution of the scenario task, and sends the process data to the data collection device; The data collection device collects the process data collected by the data collection terminal to obtain scene task data corresponding to at least one scene task; the scene task data corresponding to each of the scene tasks is used to train an artificial intelligence AI model, and the trained AI model is used to control the humanoid robot to perform scene tasks.

Images