Data Acquisition System, Terminal and Method for End Effector of Humanoid Robot

Abstract

An embodiment of the present application provides a data acquisition system, a terminal, and a method for an end effector of a humanoid robot. The system includes: at least one data acquisition terminal and a data collection device; wherein the data collection device is respectively connected to each data acquisition terminal; the data acquisition terminal has an actuator corresponding to the end effector of the humanoid robot; the data acquisition terminal is configured to collect process data corresponding to a scene task during the execution of the scene task, and send the process data to the data collection device; the data collection device is configured to collect the process data to obtain scene task data corresponding to at least one scene task, and the scene task data corresponding to each scene task is used to train an artificial intelligence AI model, and the trained AI model is used to control the humanoid robot to execute the scene task. The above system can complete data acquisition without relying on the robot body, and has strong adaptability.

Description

Technical Field

[0001] This application relates to robot technology, and in particular to a data acquisition system, a terminal, and a method for an end effector of a humanoid robot. Background Art

[0002] Embodied data plays a key role in the training of the robot embodied intelligence model and is the core foundation for realizing robot embodied intelligence. Currently, the commonly used methods for collecting robot embodied data mainly rely on teleoperation technology. Common teleoperation methods include teleoperation based on virtual reality (VR), teleoperation using motion capture technology, and teleoperation by drag teaching. These methods can effectively capture the motion characteristics and behavior patterns of the robot and provide rich data support for the training of the embodied intelligence model.

[0003] Teleoperation is strictly corresponding to the robot body. Different types of robots rely on specific teleoperation methods to be controlled by human operators to complete corresponding tasks through remote operation. However, the embodied data collected through the teleoperation method is only applicable to a specific robot body and lacks the generalization ability across robot platforms, with poor adaptability. Summary of the Invention

[0004] Embodiments of this application provide a data acquisition system, a terminal, and a method for an end effector of a humanoid robot to achieve a stronger adaptability effect.

[0005] In a first aspect, embodiments of this application provide a data acquisition system for an end effector of a humanoid robot, including:

[0006] At least one data acquisition terminal and a data collection device;

[0007] Wherein, the data collection device is respectively connected to each of the data acquisition terminals; the data acquisition terminal has an actuator corresponding to the end effector of the humanoid robot;

[0008] The data acquisition terminal is configured to collect process data corresponding to the scenario task during the execution of the scenario task and send the process data to the data collection device;

[0009] The data collection device is configured to collect the process data collected by the data acquisition terminals to obtain scenario task data corresponding to at least one scenario task; the scenario task data corresponding to each scenario task is used to train an artificial intelligence (AI) model, and the trained AI model is used to control the humanoid robot to execute the scenario task.

[0010] In a possible implementation manner, the system further includes:

[0011] A training device connected to the data collection device, the training device being used to train the AI model according to the scenario task data corresponding to each scenario task.

[0012] In a possible implementation manner, the data collection device is further used for:

[0013] Determine a data collection mode and a data collection terminal corresponding to the data collection mode;

[0014] Send a first indication message to the data collection terminal corresponding to the data collection mode, the first indication message being used to instruct the data collection terminal to execute an initialization process;

[0015] The data collection terminal is further used to send a response message corresponding to the first indication message to the data collection device after completing the initialization process.

[0016] In a possible implementation manner, the data collection device is further used to send a second indication message to the data collection terminal when the response message is a successful response message, the second indication message being used to instruct the data collection terminal to start collecting data.

[0017] In a possible implementation manner, the process data includes at least one of the following: the position information of the data collection terminal, the attitude information of the data collection terminal, the opening and closing state information of the actuator of the data collection terminal, and scene image data.

[0018] In a possible implementation manner, the data collection terminal is specifically used for:

[0019] Determine the initial position information and initial attitude information of the data collection terminal according to a plurality of scene images collected by the data collection terminal;

[0020] Obtain the position information and attitude information of the origin of the reference coordinate system according to the position information and attitude information of a preset position in the scene where the data collection terminal is located;

[0021] Establish a mapping relationship between the initial position information and the position information of the origin, and between the initial attitude information and the attitude information of the origin;

[0022] According to the mapping relationship, convert the initial position information and the initial attitude information into the position information and attitude information in the reference coordinate system respectively.

[0023] In a possible implementation manner, the data collection terminal is specifically used for:

[0024] Extract key feature points in each of the scene images, and match the key feature points in each of the scene images to obtain the spatial positions of the key feature points in different scene images;

[0025] Determine the initial position information and initial attitude information of the data acquisition terminal according to the spatial positions of the key feature points in different scene images.

[0026] In a second aspect, an embodiment of the present application provides a data acquisition terminal, including:

[0027] An actuator, an opening and closing control component, an image acquisition component, and a processing component;

[0028] Wherein, the opening and closing control component is connected to the actuator, and the opening and closing control component is used to control the opening and closing degree of the actuator;

[0029] The actuator is used to perform corresponding actions under the control of the opening and closing control component;

[0030] The image acquisition component is connected to the processing component, and the image acquisition component is used to acquire scene image data during the execution of the scene task by the data acquisition terminal;

[0031] The processing component is used to obtain process data corresponding to the scene task, and the process data includes at least one of the following: the position information of the data acquisition terminal, the attitude information of the data acquisition terminal, the opening and closing state information of the actuator, and the scene image data.

[0032] In a possible implementation manner, the processing component is further used to:

[0033] Determine the initial position information and initial attitude information of the image acquisition component according to multiple scene images acquired by the image acquisition component;

[0034] Obtain the position information and attitude information of the origin of the reference coordinate system according to the position information and attitude information of the preset position in the scene where the data acquisition terminal is located;

[0035] Establish a mapping relationship between the initial position information and the position information of the origin, and between the initial attitude information and the attitude information of the origin;

[0036] According to the mapping relationship, convert the initial position information and the initial attitude information into the position information and attitude information in the reference coordinate system respectively.

[0037] In a possible implementation manner, the processing component is specifically used to:

[0038] Extract key feature points in each of the scene images, and match the key feature points in each of the scene images to obtain the spatial positions of the key feature points in different scene images;

[0039] Determine the initial position information and initial attitude information of the image acquisition component according to the spatial positions of the key feature points in different scene images.

[0040] In a possible implementation manner, the processing component is further configured to:

[0041] Determine the initial position information and initial attitude information of the actuator according to the pose mapping relationship between the image acquisition component and the actuator and the initial position information and initial attitude information of the image acquisition component.

[0042] In a possible implementation manner, the structure of the actuator adopts at least one of the following: a two-finger gripper structure, a three-finger gripper structure, or a dexterous hand structure.

[0043] In a third aspect, an embodiment of the present application provides a method for collecting data of an end effector of a humanoid robot, which is applied to the system in the first aspect and / or various possible implementation manners of the first aspect above. The method includes:

[0044] During the execution of the scene task by the data acquisition terminal, collect the process data corresponding to the scene task, and send the process data to the data collection device;

[0045] The data collection device collects the process data collected by the data acquisition terminal to obtain scene task data corresponding to at least one scene task; the scene task data corresponding to each scene task is used to train an artificial intelligence AI model, and the trained AI model is used to control the humanoid robot to execute the scene task.

[0046] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer execution instructions are stored, and when the computer execution instructions are executed by a processor, they are used to implement the method in the third aspect above.

[0047] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which when executed by a processor implements the method in the third aspect above.

[0048] The data acquisition system, terminal and method for the end effector of a humanoid robot provided by an embodiment of the present application. The system includes: at least one data acquisition terminal and a data collection device; wherein, the data collection device is respectively connected to each data acquisition terminal; the data acquisition terminal is used to collect process data corresponding to a scenario task during the execution of the scenario task, and send the process data to the data collection device; since the data acquisition terminal has an actuator corresponding to the end effector of the humanoid robot, the data collected by the data acquisition terminal does not depend on the humanoid robot body, and the data acquisition terminal matches the humanoid robot with the corresponding end effector. Therefore, the collected data can meet the training requirements of the humanoid robot model and has strong adaptability. Further, the data collection device is used to collect the process data collected by the data acquisition terminal to obtain scenario task data corresponding to at least one scenario task; the scenario task data corresponding to each scenario task is used to train an artificial intelligence AI model, and the trained AI model is used to control the humanoid robot to execute the scenario task. Therefore, the above solution provides rich data support for the training of the robot AI model and has a high data acquisition efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0050] Figure 1 It is a schematic structural diagram of the data acquisition system for the end effector of the humanoid robot provided by the present application;

[0051] Figure 2 It is a schematic diagram of the implementation principle of the data acquisition system for the end effector of the humanoid robot provided by the present application;

[0052] Figure 3 It is a schematic flow diagram of the data acquisition system for the end effector of the humanoid robot provided by the present application Figure 1 ;

[0053] Figure 4 It is a schematic flow diagram of the data acquisition system for the end effector of the humanoid robot provided by the present application Figure 2 ;

[0054] Figure 5 It is a schematic flow diagram of the data acquisition system for the end effector of the humanoid robot provided by the present application Figure 3 ;

[0055] Figure 6 It is a schematic structural diagram of the data acquisition terminal provided by the present application Figure 1 ;

[0056] Figure 7Structural schematic diagram of the data acquisition terminal provided for this application Figure 2 ;

[0057] Figure 8 Flow schematic diagram of the data acquisition method provided for this application Figure 1 。

[0058] Through the above-mentioned drawings, specific embodiments of this application have been shown, and there will be more detailed descriptions hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of this application in any way, but to illustrate the concept of this application to those skilled in the art by referring to specific embodiments. Detailed implementation manners

[0059] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with this application. On the contrary, they are merely examples of devices and methods consistent with some aspects of this application as detailed in the appended claims.

[0060] First, the application scenarios involved in this application will be introduced:

[0061] The specific application scenario of this application is for humanoid robots, general humanoid robots, or embodied robots, to collect process data of the end effector when performing scenario tasks, such as including: position information, attitude information, opening and closing state information of the end effector, or scenario image data, etc.

[0062] Combined with the above scenarios, in the prior art, the collected embodied data is only applicable to a specific robot body, lacking the generalization ability across robot platforms and having poor adaptability.

[0063] The data acquisition system provided by this application can collect data during the execution of scenario tasks through a data acquisition terminal with an actuator corresponding to the end effector of a humanoid robot, and can adapt to different robot bodies, with strong adaptability.

[0064] Next, specific embodiments will be used to describe in detail the technical solution of this application and how the technical solution of this application solves the above technical problems. These several specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. Next, the embodiments of this application will be described with reference to the drawings.

[0065] Figure 1 Structural schematic diagram of the data acquisition system for the end effector of the humanoid robot provided for this application, as Figure 1 shown, this system includes:

[0066] At least one data acquisition terminal 100 and a data collection device 200;

[0067] Wherein, the data collection device is respectively connected to each of the data acquisition terminals; the data acquisition terminal has an actuator corresponding to the end effector of the humanoid robot;

[0068] The data acquisition terminal is configured to collect process data corresponding to the scenario task during the execution of the scenario task, and send the process data to the data collection device;

[0069] The data collection device is configured to collect the process data collected by the data acquisition terminal to obtain scenario task data corresponding to at least one scenario task; the scenario task data corresponding to each scenario task is used to train an artificial intelligence AI model, and the trained AI model is used to control the humanoid robot to execute the scenario task.

[0070] Specifically, the data acquisition terminal has an actuator corresponding to the end effector of the humanoid robot. For example, the actuator of the data acquisition terminal needs to have the same opening and closing method and similar shape as the end effector of the humanoid robot. For example, if the humanoid robot has a two-finger gripper, then the actuator of the data acquisition terminal also adopts a similar two-finger gripper structure. For example, a lightweight gripper structure can be constructed by 3D printing.

[0071] Optionally, the structure of the actuator of the data acquisition terminal adopts at least one of the following: a two-finger gripper structure, a three-finger gripper structure, or a dexterous hand structure.

[0072] During the execution of the scenario task, the data acquisition terminal collects process data corresponding to the scenario task and sends the collected process data to the data collection device.

[0073] Optionally, the process data includes at least one of the following: the position information of the data acquisition terminal, the attitude information of the data acquisition terminal, the opening and closing state information of the actuator of the data acquisition terminal, and scenario image data.

[0074] Optionally, the position information of the data acquisition terminal can be represented as a position trajectory.

[0075] Among them, the opening and closing state information of the actuator includes, for example: the opening and closing state (such as the open or closed state), the degree of opening and closing, and other information.

[0076] Optionally, the process data has corresponding timestamp information.

[0077] Optionally, the data acquisition terminal sends the collected process data to the data collection device through a wired or wireless connection.

[0078] Furthermore, the data collection device integrates the collected process data to obtain scenario task data corresponding to different scenario tasks. The scenario task data corresponding to each scenario task is used to further train the AI model, and the trained AI model will be able to drive the robot to complete the corresponding scenario task.

[0079] Optionally, the data acquisition terminal can be a handheld terminal, which is convenient for completing scenario tasks and collecting data.

[0080] The data acquisition system provided by the embodiments of the present application includes: at least one data acquisition terminal and a data collection device; wherein, the data collection device is respectively connected to each data acquisition terminal; the data acquisition terminal is used to collect process data corresponding to the scenario task during the execution of the scenario task and send the process data to the data collection device; since the data acquisition terminal has an actuator corresponding to the end effector of the humanoid robot, the data collected by the data acquisition terminal does not depend on the humanoid robot body, and the data acquisition terminal matches the humanoid robot with the corresponding end effector. Therefore, the collected data can meet the training requirements of the humanoid robot model and has strong adaptability. Furthermore, the data collection device is used to collect the process data collected by the data acquisition terminal to obtain scenario task data corresponding to at least one scenario task; the scenario task data corresponding to each scenario task is used to train the artificial intelligence AI model, and the trained AI model is used to control the humanoid robot to execute the scenario task. Therefore, the above solution provides rich data support for the training of the robot AI model and has high data acquisition efficiency.

[0081] In some embodiments, as Figure 2 shown, the system further includes:

[0082] A training device connected to the data collection device, and the training device is used to train the AI model according to the scenario task data corresponding to each scenario task.

[0083] Specifically, the data collection device sends the scenario task data corresponding to each scenario task to the training device; the training device trains the artificial intelligence AI model according to the scenario task data corresponding to each scenario task, and the trained AI model is used to control the humanoid robot to execute the scenario task.

[0084] In the above implementation manner, the training device trains the AI model based on the scenario task data corresponding to each scenario task collected by the data collection device, so that the trained AI model can control the humanoid robot to execute the corresponding scenario task, improving the reliability and accuracy of the humanoid robot in executing tasks.

[0085] In some embodiments, the data collection device is further configured to:

[0086] Determine a data collection mode and a data collection terminal corresponding to the data collection mode;

[0087] Send a first indication message to the data collection terminal corresponding to the data collection mode, where the first indication message is used to instruct the data collection terminal to perform an initialization process;

[0088] The data collection terminal is further configured to, after completing the initialization process, send a response message corresponding to the first indication message to the data collection device.

[0089] Specifically, in response to a user's instruction, determine a data collection mode. For example, the data collection mode includes at least one of the following: single-arm mode, single-arm plus top camera mode, two-arm collaboration mode, two-arm collaboration with top camera mode. Among them, the single-arm mode means using one data collection terminal, the single-arm plus top camera mode means one data collection terminal plus one head-mounted camera, the two-arm collaboration mode means two data collection terminals cooperate to collect data, and the two-arm collaboration with top camera mode means two data collection terminals cooperate and add one head-mounted camera.

[0090] After determining the data collection mode, determine the corresponding data collection terminal and send a first indication message to the corresponding data collection terminal. The first indication message is used to instruct the data collection terminal to perform an initialization process, such as initializing the reference coordinate system, converting the position and attitude determined by the data collection terminal to the position and attitude in the reference coordinate system, etc.

[0091] After the data collection terminal completes the initialization process, it sends a response message corresponding to the first indication message to the data collection device, such as indicating whether the initialization process is completed.

[0092] In some embodiments, the data collection device is further configured to, when the response message is a successful response message, send a second indication message to the data collection terminal, where the second indication message is used to instruct the data collection terminal to start collecting data.

[0093] Specifically, when the data collection device receives a successful response message sent by the data collection terminal, it sends a second indication message to the data collection terminal to instruct the data collection terminal to start collecting data. That is, the data collection terminal executes the scenario task and performs data collection.

[0094] In the above embodiments, the data collection terminal and the data collection device interact with each other to complete the initialization process before data collection, with low complexity and high efficiency.

[0095] In some embodiments, the initialization process is as follows:

[0096] Based on multiple scene images collected by the data acquisition terminal, determine the initial position information and initial attitude information of the data acquisition terminal;

[0097] Based on the position information and attitude information of a preset position in the scene where the data acquisition terminal is located, obtain the position information and attitude information of the origin of the reference coordinate system;

[0098] Establish a mapping relationship between the initial position information and the position information of the origin, and between the initial attitude information and the attitude information of the origin;

[0099] According to the mapping relationship, convert the initial position information and initial attitude information into the position information and attitude information in the reference coordinate system respectively.

[0100] Specifically, obtain the collected multiple scene images, and determine the initial position information and initial attitude information of the data acquisition terminal based on the multiple scene images, that is, the initial position information and initial attitude information in the camera coordinate system;

[0101] Obtain the position information and attitude information of a preset position in the scene where the data acquisition terminal is located, and use the position information and attitude information of this preset position as the position information and attitude information of the origin of the reference coordinate system.

[0102] Establish a mapping relationship between the initial position information of the data acquisition terminal and the position information of the origin, and between the initial attitude information and the attitude information of the origin, and according to this mapping relationship, convert the initial position information of the data acquisition terminal into the position information in the reference coordinate system, and convert the initial attitude information of the data acquisition terminal into the attitude information in the reference coordinate system.

[0103] Optionally, during the execution of the scene task by the data acquisition terminal, continuously obtain scene images, determine the real-time position information and attitude information of the data acquisition terminal based on the scene images, and based on the above mapping relationship, can be converted into the real-time position information and attitude information in the reference coordinate system.

[0104] In the above embodiments, by converting the position information and attitude information of the data acquisition terminal into the position information and attitude information in the reference coordinate system, it makes the data more matching with the scene and the collected data more in line with the actual requirements.

[0105] Exemplarily, as Figure 3 shown, the data acquisition workflow in single-arm mode is as follows:

[0106] The single-arm mode requires the use of a handheld data acquisition terminal, which is connected to the data collection device. Positioning QR codes are posted in the data acquisition scenario. The positioning QR codes contain the position information and attitude information of the current position to ensure the continuity of the position of the data acquisition terminal.

[0107] 1. Select mode:

[0108] Select "single-arm mode" in the data collection device and specify the corresponding data acquisition terminal. The actuator of the data acquisition terminal needs to have the same opening and closing method and a similar shape as the corresponding robot end effector. For example, if the corresponding robot is a two-finger gripper, then the actuator of the data acquisition terminal also needs to be a similar two-finger gripper structure.

[0109] 2. Initialize the coordinate system:

[0110] Click the "Prepare" button in the data collection device, which sends the first instruction message to the data acquisition terminal. After receiving the first instruction message, the data acquisition terminal records the position information and attitude information of the current actuator (such as the gripper), and records them as the initial position and initial attitude. The user (or operator) scans the positioning QR code through the handheld data acquisition terminal to perform the coordinate system initialization process. After scanning the positioning QR code, the position and attitude of the positioning QR code are determined as the position and attitude of the origin of the reference coordinate system for data recording, and the reference coordinate system for data recording is established, and the mapping relationship between the position and attitude of the origin and the initial position and initial attitude of the actuator is calculated. After completing the initialization of the reference coordinate system for data recording, the data acquisition terminal sends a response message to the data collection device to indicate whether the initialization process is completed. The data collection device can send a second instruction message (i.e., the start instruction) to instruct the data acquisition terminal to start collecting data.

[0111] 3. Execute the task:

[0112] After the data collection device sends the second instruction message (i.e., the start instruction), the operator completes the corresponding scenario tasks through the data acquisition terminal.

[0113] During the task execution process, the data acquisition terminal continuously records the position information, attitude information, opening and closing state information of the end effector, the scene image data captured by the camera, and the corresponding timestamp information.

[0114] The data is sent to the data collection device through a wired or wireless connection, and the data collection device continuously records the process data with timestamps.

[0115] 4. End data recording:

[0116] After completing the scenario task, click the "Complete" button on the handheld data collection terminal, which sends an end command to the data collection device. The data collection device stops recording and packs all the collected data into scenario task data corresponding to a scenario task, and sends the scenario task data to the training device.

[0117] 5. Model Training:

[0118] After receiving scenario task data corresponding to a cumulative number of scenario tasks exceeding a preset quantity, input these scenario task data into the training device to further train the AI model. The trained model will be able to drive the robot to complete the corresponding scenario task.

[0119] Exemplarily, as Figure 2 、 Figure 4 shown, the data collection workflow for the dual-arm collaborative mode with a top camera is as follows:

[0120] This mode requires two handheld data collection terminals and a head-mounted camera, which are connected to the data collection device. Positioning QR codes are posted in the data collection scenario to ensure the continuity of the positions of the data collection terminals.

[0121] 1. Select Mode:

[0122] Select the "Dual-arm Collaborative Mode with a Top Camera" in the data collection device, specify the corresponding two data collection terminals, and connect the camera to the data collection terminal and the data collection device. The actuators of the data collection terminals need to have the same opening and closing method and the same shape as the corresponding end effectors of the robot.

[0123] 2. Initialize the Coordinate System:

[0124] Click the "Prepare" button in the data collection device, which sends a first indication message to the two data collection terminals. After receiving the first indication message, the data collection terminals record the position information and attitude information of the current actuator (such as a gripper), and record them as the initial position and initial attitude. The user scans the positioning QR code through the handheld data collection terminal to perform the coordinate system initialization process. After scanning the positioning QR code, the position and attitude of the positioning QR code are determined as the position and attitude of the origin of the reference coordinate system for data recording, and the reference coordinate system for data recording is established. After completing the initialization of the reference coordinate system for data recording and ensuring that there is image data transmission from the head-mounted camera, the data collection terminal sends a response message to the data collection device to indicate whether the initialization process is completed, that is, whether it is ready. The data collection device can send a second indication message (i.e., the start command) to indicate that the data collection terminal starts to collect data.

[0125] 3. Execute Task:

[0126] After the data collection device issues the second instruction message (i.e., the start command), the operator completes the corresponding scenario task through the data acquisition terminal, and at the same time requires the head-mounted camera to capture the operation process of at least one handheld data acquisition terminal, that is, at least one handheld data acquisition terminal needs to be kept within the frame of the head-mounted camera.

[0127] During the task execution, the data acquisition terminal continuously records and transmits process data, including at least one of the following: position information, attitude information, opening and closing state information of the end effector, scene image data captured by the camera of the data acquisition terminal, and corresponding timestamp information. The head-mounted camera continuously records and transmits scene image data with timestamps.

[0128] The process data is sent to the data collection device through a wired or wireless connection, and the data collection device continuously records the process data with timestamps.

[0129] 4. End data recording:

[0130] After completing the scenario task, click the "Complete" button on the handheld data acquisition terminal, that is, send an end command to the data collection device. The data collection device stops recording and packages all the collected data into scenario task data corresponding to one scenario task, and sends the scenario task data to the training device.

[0131] 5. Model training:

[0132] When receiving the scenario task data corresponding to more than a preset number of scenario tasks accumulated, these scenario task data are input into the training device to further train the AI model. The trained model will be able to drive the robot to complete the corresponding scenario task.

[0133] In some embodiments, the data acquisition terminal is specifically configured to:

[0134] Extract key feature points in each of the scene images, and match the key feature points in each of the scene images to obtain the spatial positions of the key feature points in different scene images;

[0135] Determine the initial position information and initial attitude information of the data acquisition terminal according to the spatial positions of the key feature points in different scene images.

[0136] Specifically, scene images are obtained through the image acquisition component of the data acquisition terminal, for example, obtaining multiple consecutive scene image frames.

[0137] Extract key feature points in multiple scene image frames, such as corner points, edges or textures; match the key feature points extracted from consecutive scene image frames to determine the spatial positions of the same key feature points in different scene image frames.

[0138] Based on the spatial positions of the matched key feature points in different scene image frames, use the Perspective-n-Point (PNP) algorithm to estimate the position and pose of the current image acquisition component. Optionally, use the position and pose of this image acquisition component as the position and pose of the data acquisition terminal.

[0139] Optionally, after calculating the position and pose for the first time, continuously repeat this process to update the real-time position and pose of the data acquisition terminal.

[0140] Exemplarily, as Figure 5 shown, the initialization process is as follows:

[0141] S1. Input consecutive image frames;

[0142] S2. Extract key feature points in the image frames;

[0143] S3. Match the extracted feature points to determine the spatial positions of the same feature points in different image frames;

[0144] S4. Use the PNP algorithm to estimate the position and pose of the current camera;

[0145] S5. Whether a positioning QR code is scanned;

[0146] S6. Coordinate system mapping; that is, establish the mapping relationship between the position and pose of the data acquisition terminal in the camera coordinate system and the position and pose of the origin in the reference coordinate system.

[0147] S7. Output the position and pose of the current camera in the reference coordinate system.

[0148] Wherein, the camera refers to the image acquisition component in the data acquisition terminal.

[0149] Figure 6 Schematic diagram of the data acquisition terminal structure of the end effector of the humanoid robot provided by this application Figure 1 as Figure 6 shown, this data acquisition terminal includes:

[0150] Actuator 10, opening and closing control component 11, image acquisition component 12 and processing component (not shown in the figure);

[0151] Wherein, the opening and closing control component 11 is connected to the actuator 10, and the opening and closing control component 11 is used to control the opening and closing degree of the actuator 10;

[0152] The actuator 10 is configured to perform corresponding actions under the control of the opening / closing control component 11;

[0153] The image acquisition component 12 is connected to the processing component, and is configured to acquire scene image data of the data acquisition terminal during the execution of the scene task;

[0154] The processing component is configured to obtain process data corresponding to the scene task, where the process data includes at least one of the following: the position information of the data acquisition terminal, the attitude information of the data acquisition terminal, the opening / closing state information of the actuator, and the scene image data.

[0155] Specifically, Figure 6 、 Figure 7 The structure of the data acquisition terminal shown in is only an example, and the shapes and connections of each component can also adopt other ways, which are not limited in the embodiments of the present application.

[0156] The data acquisition terminal has an actuator corresponding to the end effector of the humanoid robot. For example, the actuator of the data acquisition terminal needs to have the same opening / closing method and a similar shape as the end effector of the humanoid robot. For example, if the end effector of the humanoid robot is a two-finger gripper, then the actuator of the data acquisition terminal also adopts a similar two-finger gripper structure.

[0157] Optionally, the structure of the actuator of the data acquisition terminal adopts at least one of the following: a two-finger gripper structure, a three-finger gripper structure, or a dexterous hand structure.

[0158] The opening / closing control component is configured to control the opening / closing action of the actuator, that is, to control the actuator to perform corresponding actions.

[0159] During the execution of the scene task, the data acquisition terminal controls the actuator to perform operations corresponding to the scene task through the opening / closing control component, the image acquisition component acquires scene image data during the execution of the scene task, and the processing component records the process data during the execution of the scene task, such as including at least one of the following: the position information of the data acquisition terminal, the attitude information, the opening / closing state information, the scene image data, etc.

[0160] Optionally, the process data includes at least one of the following: the position information of the data acquisition terminal, the attitude information of the data acquisition terminal, the opening / closing state information of the actuator of the data acquisition terminal, and the scene image data.

[0161] Optionally, the position information of the data acquisition terminal can be represented as a position trajectory.

[0162] Among them, the opening and closing state information of the actuator includes, for example: the opening and closing state (such as open or closed), the degree of opening and closing, and other information.

[0163] Optionally, the process data has corresponding timestamp information.

[0164] Optionally, the data acquisition terminal further includes: a switch button for controlling the startup or shutdown of the data acquisition terminal.

[0165] Optionally, the data acquisition terminal further includes: a power supply component for supplying power to the data acquisition terminal.

[0166] For the data acquisition terminal provided in this embodiment, its implementation principle and technical effects are similar to those of the data acquisition system, and will not be elaborated here.

[0167] In some embodiments, the data acquisition terminal can implement the initialization process in the following manner, that is, the processing component is further configured to:

[0168] Determine the initial position information and initial attitude information of the image acquisition component according to multiple scene images acquired by the image acquisition component;

[0169] Obtain the position information and attitude information of the origin of the reference coordinate system according to the position information and attitude information of the preset position in the scene where the data acquisition terminal is located;

[0170] Establish a mapping relationship between the initial position information and the position information of the origin, and between the initial attitude information and the attitude information of the origin;

[0171] According to the mapping relationship, respectively convert the initial position information and the initial attitude information into the position information and attitude information in the reference coordinate system.

[0172] In some embodiments, the processing component is specifically configured to:

[0173] Extract key feature points in each of the scene images, and match the key feature points in each of the scene images to obtain the spatial positions of the key feature points in different scene images;

[0174] Determine the initial position information and initial attitude information of the image acquisition component according to the spatial positions of the key feature points in different scene images.

[0175] In some embodiments, the processing component is further configured to:

[0176] Determine the initial position information and initial attitude information of the actuator according to the pose mapping relationship between the image acquisition component and the actuator and the initial position information and initial attitude information of the image acquisition component.

[0177] Specifically, the actuator is fixedly connected to the image acquisition component (such as a camera). Therefore, the coordinate transformation matrix between the pose of the image acquisition component and the pose of the actuator can be obtained, and a pose mapping relationship is established.

[0178] Furthermore, based on the pose mapping relationship, as well as the initial position information and initial pose information of the image acquisition component, the initial position information and initial pose information of the actuator are determined.

[0179] Optionally, the process data may include: the position information and pose information of the image acquisition component and / or the actuator. Alternatively, the position information and pose information of the data acquisition terminal can be obtained based on the position information and pose information of the image acquisition component and the actuator.

[0180] In summary, in the embodiments of the present application, by using the data acquisition terminal, the position information, pose information, opening and closing state information of the end effector of the robot, and relevant image data are collected. The task model is trained through the AI algorithm, and the embodied intelligent agent is controlled to complete the scene task through the trained model. Compared with the traditional embodied data acquisition scheme, the data acquisition can be completed without relying on the robot body through the data acquisition terminal, which reduces the threshold of data acquisition. At the same time, due to the more flexible handheld operation, the diversity of the embodied data is greatly improved. By replacing the actuator part of the data acquisition terminal, the replacement of end effectors such as two-finger grippers, three-finger grippers, or dexterous hands can be completed, which can adapt to multiple robots and has strong adaptability. Since the model is trained with scene task data, the inverse kinematics can be solved for different robot bodies, and the same model can be adapted to different robot bodies, which has strong adaptability.

[0181] In some embodiments, the data acquisition terminal may further include: at least one processor and a memory. Among them, components such as the processor, memory, and image acquisition component are connected through a bus.

[0182] In the specific implementation process, at least one processor executes the computer execution instructions stored in the memory, so that at least one processor executes the data acquisition process of the above data acquisition terminal.

[0183] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU for short), or other general-purpose processors, digital signal processors (DSP for short), application specific integrated circuits (ASIC for short), etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in combination with the invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.

[0184] The memory may include a random access memory (RAM), and may also include a non-volatile memory (NVM), such as at least one disk memory.

[0185] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, the buses in the drawings of this application are not limited to only one bus or one type of bus.

[0186] Figure 8 Flow schematic of the data acquisition method provided for this application Figure 1 , the method of this embodiment is applied to the system in any of the above embodiments, such as Figure 8 shown, the method includes:

[0187] S801. During the execution of the scenario task by the data acquisition terminal, collect the process data corresponding to the scenario task, and send the process data to the data collection device;

[0188] S802. The data collection device collects the process data collected by the data acquisition terminal to obtain the scenario task data corresponding to at least one scenario task; the scenario task data corresponding to each scenario task is used to train an artificial intelligence (AI) model, and the trained AI model is used to control the humanoid robot to execute the scenario task.

[0189] The method provided in this embodiment can be applied to the above system or terminal embodiment, and its implementation principle and technical effects are similar. Therefore, details are not described herein.

[0190] This application also provides a computer program product, including a computer program which, when executed by a processor, implements the above method.

[0191] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above method.

[0192] The above-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk. The readable storage medium can be any available medium accessible by a general-purpose or special-purpose computer.

[0193] An exemplary readable storage medium is coupled to the processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in a device.

[0194] The division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Additionally, the couplings, direct couplings, or communication connections shown or discussed among each other can be indirect couplings or communication connections through some interfaces, devices, or units, and can be in electrical, mechanical, or other forms.

[0195] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0196] In addition, in each embodiment of the present invention, each functional unit may be integrated into a processing unit, may exist physically alone for each unit, or two or more units may be integrated into one unit.

[0197] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in each embodiment of the present invention. The aforementioned storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs and other various media that can store program codes.

[0198] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When this program is executed, it executes the steps including the above method embodiments; and the aforementioned storage medium includes: ROMs, RAMs, magnetic disks, or optical discs and other various media that can store program codes.

[0199] Finally, it should be noted that: After considering the specification and practicing the invention disclosed herein, those skilled in the art will readily think of other implementation schemes of the present invention. The present invention aims to cover any variations, uses, or adaptive changes of the present invention. These variations, uses, or adaptive changes follow the general principles of the present invention and include common general knowledge or conventional technical means in the technical field not disclosed in the present invention. It is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present invention is only limited by the appended claims.

Claims

1. A data acquisition system for an end effector of a humanoid robot, characterized in that: include: At least one data collection terminal, a data collection device and a head-mounted camera; the data collection device is respectively connected to each of the data collection terminals; The data acquisition terminal has an actuator corresponding to the end effector of the humanoid robot; The data acquisition terminal comprises: an image acquisition component and a processing component; the image acquisition component is connected to the processing component, and the image acquisition component is used to acquire scene image data of the data acquisition terminal during the execution of a scene task; the processing component is used to obtain process data corresponding to the scene task based on the scene image data, and the process data comprises at least one of the following: position information and posture information of the data acquisition terminal in a reference coordinate system, opening and closing state information of the actuator, and the scene image data; The data acquisition terminal is used to send the process data to the data collection device; The data collection device is used to collect the process data to obtain scene task data corresponding to at least one scene task; the scene task data corresponding to each of the scene tasks is used to train an artificial intelligence AI model, and the trained AI model is used to control the humanoid robot to perform the scene task; The data collection device is also used to determine a data collection mode and a data collection terminal corresponding to the data collection mode; wherein the data collection mode is a single-arm mode or a single-arm plus a head-mounted camera mode or a dual-arm collaborative mode or a dual-arm collaborative plus a head-mounted camera mode, the single-arm mode refers to the use of one data collection terminal, the single-arm plus a head-mounted camera mode refers to one data collection terminal plus a head-mounted camera, the dual-arm collaborative mode refers to two data collection terminals cooperating to collect data, and the dual-arm collaborative with head-mounted camera mode refers to two data collection terminals cooperating and adding a head-mounted camera; when the data collection mode corresponds to two data collection terminals, the two data collection terminals are both connected to the data collection device; a first indication message is sent to the data collection terminal corresponding to the data collection mode, and the first indication message is used to instruct the data collection terminal to perform an initialization process; The data acquisition terminal is further used to send response information corresponding to the first indication information to the data collection device after completing the initialization process; The data collection device is further used to send second indication information to the data collection terminal when the response information is successful response information, and the second indication information is used to instruct the data collection terminal to start collecting data.

2. The system according to claim 1, characterized in that The system further comprises: A training device connected to the data collection device, wherein the training device is used to train the AI ​​model according to the scenario task data corresponding to each of the scenario tasks.

3. The system according to claim 2, characterized in that The processing component of the data acquisition terminal is specifically used for: Determining initial position information and initial posture information of the data acquisition terminal according to a plurality of scene images acquired by the data acquisition terminal; Obtaining the position information and attitude information of the origin of the reference coordinate system according to the position information and attitude information of the preset position in the scene where the data acquisition terminal is located; Establishing a mapping relationship between the initial position information and the position information of the origin, and between the initial posture information and the posture information of the origin; According to the mapping relationship, the initial position information and the initial posture information are respectively converted into the position information and the posture information in the reference coordinate system.

4. The system according to claim 3, characterized in that The processing component of the data acquisition terminal is specifically used for: Extracting key feature points from each of the scene images, and matching the key feature points in each of the scene images to obtain the spatial position of each of the key feature points in different scene images; The initial position information and initial posture information of the data acquisition terminal are determined according to the spatial position of each of the key feature points in the different scene images.

5. A data collection method for an end effector of a humanoid robot, characterized in that: Applied to the system according to any one of claims 1 to 4, the method comprises: The data collection device determines a data collection mode and a data collection terminal corresponding to the data collection mode, and sends first indication information to the data collection terminal corresponding to the data collection mode, wherein the first indication information is used to instruct the data collection terminal to perform an initialization process; when the data collection mode corresponds to two data collection terminals, the two data collection terminals are both connected to the data collection device; The data acquisition terminal, after completing the initialization process, sends response information corresponding to the first indication information to the data collection device; The data collection device sends second instruction information to the data collection terminal when the response information is successful response information, and the second instruction information is used to instruct the data collection terminal to start collecting data; The data acquisition terminal sends the process data to the data collection device; the process data includes: position information and posture information of the data acquisition terminal in the reference coordinate system, and opening and closing state information of the actuator of the data acquisition terminal and scene image data; The data collection device collects the process data to obtain scene task data corresponding to at least one scene task; the scene task data corresponding to each of the scene tasks is used to train an artificial intelligence AI model, and the trained AI model is used to control the humanoid robot to perform scene tasks.

Images