Data annotation method and device, cluster, storage medium and program product

Abstract

The invention discloses a data annotation method and device, a cluster, a storage medium and a program product, and relates to the field of data annotation. The computing device obtains target data, the target data comprises action data and videos at all moments in the process that the object executes the first task, and the computing device takes the moment corresponding to the action conforming to the first specified feature in the action data as the starting moment and / or the ending moment of each sub-video; a video is divided into a plurality of sub-videos, the computing device obtains respective first description information of each sub-video, the first description information is used for describing a second task executed by an object in a time period corresponding to each sub-video, and the second task is a sub-task of the first task. Therefore, the videos, the actions and the first description information corresponding to the sub-tasks of the first task are obtained, and the labeling efficiency of the target data is remarkably improved.

Description

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on March 29, 2024, with application number 202410383072.2 and application name “A data annotation method, device, storage medium and program product”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of data annotation technology, and in particular to a data annotation method, device, cluster, storage medium and program product. Background Art

[0003] Embodied artificial intelligence (EAI) refers to the acquisition of autonomous intelligence through learning from data gained from interactions between an agent and its environment. It is a key research direction in artificial intelligence and robotics. Models that can be applied to embodied intelligent robots are called embodied intelligence models. Embodied intelligence models can be trained using labeled embodied intelligence data, accelerating the evolution of embodied intelligent robots.

[0004] Currently, data annotation methods are mainly used for training traditional image processing or natural language processing related models, mainly for annotating image data or text data. These methods cannot be used to annotate embodied intelligence data. In other words, there is a lack of a data annotation method for embodied intelligence data. Summary of the Invention

[0005] The present application provides a data labeling method, device, cluster, storage medium and program product, which obtain target data based on the action data of the object at each moment during the execution of the first task, divide the video of the object performing the first task into multiple sub-videos, and then realize automatic labeling of the target data based on the first description information of each sub-video obtained, thereby improving the labeling efficiency of embodied intelligent data.

[0006] In a first aspect, the present application provides a data labeling method, which includes: a computing device obtains target data, wherein the target data includes action data and video at various moments in the process of an object performing a first task, the object performs actions according to the action data to perform the first task, and the video is a video of the object performing the first task; the computing device determines at least one moment corresponding to at least one key action data in the action data as at least one first segmentation moment, and the action performed by the object according to at least one key action data at at least one first segmentation moment meets a first specified feature; the computing device divides the video into multiple sub-videos, wherein at least one first segmentation moment is the start moment and / or end moment of each sub-video; the computing device obtains first description information of each sub-video; the first description information is used to describe a second task performed by the object in the time period corresponding to each sub-video; the second task is a subtask of the first task.

[0007] It can be understood that the computing device uses the moment corresponding to the action data in the target data that meets the first specified feature as the starting moment and / or ending moment of the sub-video, thereby dividing the video into multiple sub-videos, and the time period of each sub-video is the time period between its starting moment and ending moment, and then obtains the first description information of each sub-video, and then the computing device determines the correspondence between each sub-video, action data, and the first description information, that is, realizes automatic labeling of the target data and improves labeling efficiency.

[0008] In one possible implementation, the method also includes: the computing device outputs labeled data, the labeled data includes multiple sub-videos and first description information of each sub-video, the labeled data is used to train the embodied intelligent model, the object performs actions corresponding to the task through the embodied intelligent model, and the labeled data includes sample data and label data.

[0009] It is understandable that the computing device can output the sub-video and the corresponding first description information as labeled data for training the embodied intelligent model, and can include two types of data: sample data and label data, which can provide supervised learning data for the embodied intelligent model, improve the training efficiency of the embodied intelligent model, and improve the training effect.

[0010] In one possible implementation, if the annotation data is first annotation data, the annotation data also includes second description information, wherein each sub-video and the second description information are sample data, the second description information is used to describe the first task, and the first description information of each sub-video is label data.

[0011] It can be understood that each sub-video and the second description information serve as the first sample data in the first annotated data, and the first description information of each video serves as the first label data in the first annotated data. The first annotated data obtained in this way is used to train the embodied intelligent model to enable it to have the ability to split a complex task into multiple simple tasks when it receives it.

[0012] In one possible implementation, if the annotation data is the second annotation data, the second annotation data also includes multiple segments of sub-action data, each sub-video and the corresponding first description information are sample data in the second annotation data, and the multiple segments of sub-action data are determined as label data in the annotation data; the multiple segments of sub-action data are obtained by dividing the action data according to at least one first segmentation moment, and the at least one first segmentation moment is the starting moment and / or ending moment of each segment of sub-action data.

[0013] It can be understood that each sub-video and the corresponding first description information are used as sample data in the second annotated data, and multiple segments of action data are used as label data in the second annotated data. The second annotated data obtained in this way is used to train the embodied intelligent model to enable it to have the ability to infer action data based on the video and the first description information.

[0014] In one possible implementation, the computing device divides the video into multiple sub-videos, including: the computing device determines at least one moment corresponding to at least one key video frame in the video as at least one second segmentation moment, wherein the video content of the video meets the second specified feature at the at least one second segmentation moment; the computing device divides the video into multiple sub-videos, wherein at least one first segmentation moment or at least one second segmentation moment is the start moment and / or end moment of each sub-video.

[0015] It can be understood that the computing device determines the key video frames through the changes in the video content, obtains the second segmentation moment, and then divides the video into multiple sub-videos according to the first segmentation moment and the second segmentation moment, which can further improve the accuracy and precision of video segmentation.

[0016] In one possible implementation, the method also includes: providing an interactive interface, the interactive interface being used to display each sub-video and the first description information of each sub-video; receiving a trigger operation on an interactive control in the interactive interface, and adjusting the annotation data according to a set function corresponding to the interactive control; the setting function includes modifying the first description information, and / or the second description information, the start time, and / or the end time of each sub-video.

[0017] It can be understood that by displaying the interactive interface, the intuitiveness of the annotation data is improved, and the annotation data is adjusted according to the triggering operation of the interactive controls in the interactive interface, which can meet the different annotation needs of users and further improve the accuracy and efficiency of annotation.

[0018] In one possible implementation, if a trigger operation is received for an interactive control in an interactive interface, the annotation data is adjusted according to the set function corresponding to the interactive control, including: if a selection operation is received for the start time and end time in the interactive interface, the target data between the start time and the end time is annotated according to the first description information corresponding to each sub-video between the start time and the end time to obtain the annotation data.

[0019] It is understandable that labeling the target data within the start time and end time range selected by the user avoids generating redundant labeling data, reduces the usage of storage and computing resources, and improves labeling efficiency.

[0020] In one possible implementation, the computing device obtains the first description information of each sub-video, including: the computing device inputs each sub-video into a video description model, and obtains the first description information corresponding to each sub-video output by the video description model.

[0021] It is understandable that the computing device uses the video description model to improve the efficiency of obtaining the first description information corresponding to each sub-video, avoid relying on manual description or summary of the video content, and reduce manpower investment.

[0022] In one possible implementation, each sub-video is input into a video description model to obtain first description information corresponding to each sub-video output by the video description model, including: inputting second description information and each sub-video into the video description model to obtain first description information corresponding to each sub-video output by the video description model; the second description information is used to describe the first task performed by the object; wherein the video description model is a neural network model trained based on multiple video samples, second description information samples corresponding to each of the multiple video samples, and first description information labels.

[0023] It can be understood that the computing device inputs the second description information of the first task performed by the description object and each sub-video into the video description model, providing the video description model with the semantic center of the first description information to be output, which can improve the accuracy of the video description model when outputting the first description information.

[0024] In a second aspect, the present application provides a data labeling device, which includes a module for executing the method of the above-mentioned first aspect or any implementation method of the first aspect.

[0025] In one possible implementation, the embodiment of the present application can divide the data labeling device into functional modules according to the method provided in the first aspect above. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. Exemplarily, the embodiment of the present application can divide the data labeling device into a first acquisition module, a determination module, a division module, and a second acquisition module, etc. according to the function. The description of the possible technical solutions and beneficial effects executed by the above-mentioned divided functional modules can refer to the technical solutions provided by the above-mentioned first aspect or its corresponding possible implementation method, and will not be repeated here.

[0026] In a third aspect, an embodiment of the present application provides a computing device, which includes a processor and a memory, and the processor is coupled to the memory; the memory is used to store computer instructions, and the computer instructions are loaded and executed by the processor to enable the computing device to implement the data labeling method described in the above aspects.

[0027] In a fourth aspect, an embodiment of the present application provides a computing device cluster, which includes at least one computing device, each computing device including a processor and a memory, and the processor is coupled to the memory; the processor of at least one computing device is used to execute computer instructions stored in the memory of at least one computing device, so that the computing device cluster executes the data labeling method provided in various optional implementation methods of the above-mentioned first aspect.

[0028] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores at least one computer program instruction, and the computer program instruction is loaded and executed by a processor to implement the data labeling method described in the above aspects.

[0029] In a sixth aspect, embodiments of the present application provide a computer program product comprising computer instructions stored in a computer-readable storage medium. A processor of a computing device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computing device to perform the data labeling method provided in various optional implementations of the first aspect.

[0030] For the specific descriptions of the second to sixth aspects and their various implementations in this application, reference can be made to the detailed descriptions in the first aspect and its various implementations; and for the beneficial effects of the second to sixth aspects and their various implementations, reference can be made to the analysis of the beneficial effects in the first aspect and its various implementations, which will not be repeated here.

[0031] These and other aspects of the present application will become more readily apparent from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a flow chart of a data annotation method applied to images;

[0033] Figure 2 A schematic diagram of the architecture of a data annotation system provided in an embodiment of the present application;

[0034] Figure 3 is a schematic diagram showing an interactive interface according to an exemplary embodiment;

[0035] Figure 4 is a flow chart showing a data labeling method according to an exemplary embodiment;

[0036] Figure 5 yes Figure 4 A schematic diagram showing a comparison of an interactive interface before and after an automatic annotation process involved in the illustrated embodiment;

[0037] Figure 6 yes Figure 4 A schematic diagram of an interactive interface for displaying segmented task annotation data involved in the illustrated embodiment;

[0038] Figure 7 yes Figure 4 A schematic diagram of another interactive interface for displaying segmented task annotation data involved in the illustrated embodiment;

[0039] Figure 8 A schematic diagram of the structure of a data annotation device provided in this application;

[0040] Figure 9 is a schematic diagram of a computing device provided by an exemplary embodiment of the present application;

[0041] Figure 10 is a schematic diagram of a computing device cluster provided by an exemplary embodiment of the present application;

[0042] Figure 11 This is a schematic diagram of the connection method between computing device clusters provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0044] In this document, "plurality" refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0045] Furthermore, in the description of the embodiments of the present application, unless otherwise specified, "a plurality" refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0046] In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit differences. At the same time, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way for easy understanding.

[0047] First, the application scenarios of the embodiments of the present application are exemplarily introduced.

[0048] With the development of artificial intelligence-related technologies, various models that can be applied to visual detection, segmentation, recognition, or natural language processing (NLP) have demonstrated excellent performance in certain tests. However, these traditional neural network models focus solely on data processing and cannot actually interact with the physical world. Therefore, embodied intelligence has emerged. Compared to the aforementioned neural network models, embodied intelligence has an additional "body" that can collect environmental information through various sensors, perform physical operations using mechanical actuators (such as robotic arms and grippers), and learn and evolve through interaction with the environment. The models applied to these embodied intelligent robots are called embodied intelligence models.

[0049] Specifically, the embodied intelligence model can be divided into an embodied task planning model and an embodied command execution model. These two models can be trained through supervised learning, thereby accelerating the evolution of embodied intelligent robots. In other words, by labeling embodied task planning data and embodied command execution data (collectively referred to as embodied intelligence data), the capabilities of the embodied task planning model and the embodied command execution model can be improved, thereby enhancing the intelligence of the embodied intelligent robot. Embodied intelligence data typically includes first-person perspective video captured at various moments while the robot executes a command, as well as robot motion data. Robot motion data refers to the motion and action information involved in the robot's execution of a task. This data may include joint angles, positions, velocities, accelerations, and more. Different types of robots (e.g., industrial robots, service robots, exoskeleton robots, etc.) require different types of motion data to achieve their respective functions. For example, motion data for a lower limb exoskeleton robot can be used to control the robot's walking, while motion data for an industrial robot can be used to perform specific production tasks. Specifically, motion data may include information such as chassis speed, desired end-user posture, and gripper opening and closing.

[0050] Furthermore, in order to accelerate the evolution of embodied intelligent robots, embodied intelligence data needs to be labeled. However, traditional data labeling mainly focuses on labeling relevant data of visual detection, segmentation, and recognition models.

[0051] like Figure 1 As shown, Figure 1 This is a flow chart of a data annotation method applied to images, which includes: S101, displaying the image to be annotated; S102, if a preset event for the image to be annotated is monitored, determining the data annotation starting point and the data annotation end point according to the preset event; S103, generating and displaying a data annotation frame according to the determined data annotation starting point and the data annotation end point; S104, if a selection instruction for any data annotation frame is received, labeling the selected data annotation frame, and generating annotation data corresponding to the image to be annotated based on the labeled data annotation frame.

[0052] In this method, by monitoring preset events of the image to be annotated, where the preset events are mainly inputs from external devices, such as mouse clicks and drags, the data annotation starting point and data annotation end point in the image to be annotated are determined, and the corresponding data annotation frame is generated. By labeling the selected data annotation frame, the accuracy of labeling data such as images is improved.

[0053] Obviously, this method can only be applied to image data, and cannot be applied to embodied intelligence data. Currently, the annotation of embodied intelligence data still mainly relies on manual annotation, which is inefficient.

[0054] In view of this, the present application provides a data labeling method, which divides the video in the target data (embodied intelligent data) into multiple sub-videos according to the action data in the target data, and then labels the target data according to the first description information of each sub-video obtained to obtain labeled data, thereby realizing automatic labeling of embodied intelligent data and improving labeling efficiency.

[0055] Next, the system architecture of the embodiment of the present application is exemplarily introduced.

[0056] Figure 2 This is a schematic diagram of the architecture of a data annotation system provided in an embodiment of the present application. Figure 2 As shown, the data annotation system 100 includes a computing device 110 , a database 120 , a terminal device 130 , a data storage system 140 and a data acquisition device 150 .

[0057] The computing device 110 may be a terminal, such as a computer, a mobile phone terminal, a tablet computer, a laptop computer, a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, an extended reality (ER) device, a camera, or an in-vehicle computer, or an edge device (e.g., a box with a processing chip). In the present application, the computing device 110 may be a computing device connected to a base station, or a computing device deployed at a base station, such as a server or a cloud device.

[0058] The data acquisition device 150 is used to collect target data (or data to be annotated / raw data) and store the target data in the database 120. The data acquisition device 150 and the computing device 110 can be the same or different devices. In the embodiment of the present application, the data acquisition device 150 can be a computing device or network device with embodied intelligent data acquisition capabilities, such as an embodied intelligent robot equipped with multiple sensors, or embodied intelligent simulation software.

[0059] Combined with the data labeling system 100, the data labeling method provided in this embodiment can be applied to any scenario where target data needs to be automatically labeled, verified, and modified. The data format of the target data may include but is not limited to: rosbag data packet, where rosbag is a commonly used embodied intelligent data format, which can include the raw data of various sensors, and each data has corresponding collection time information.

[0060] It should be noted that, in actual applications, the data maintained in the database 130 may not all come from the data acquisition device 150, but may also be received from other devices. The above description should not be used as a limitation on the embodiments of the present application.

[0061] Furthermore, according to the functions performed by the computing device 110, the computing device 110 can be further divided into Figure 2 The architecture shown, such as Figure 2 As shown, the computing device 110 is configured with a computing module 111 , an I / O interface 112 , a pre-processing module 113 and a data storage module 114 .

[0062] Taking the computing device 110 as an example of a computing device connected to an access site such as a base station or a wireless access point, the computing module 111 can be a CPU, other general-purpose processors, DSP, ASIC, FPGA, GPU or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. on a computer.

[0063] For example, the calculation module 111 is used to run multiple modules, which may specifically include: input module 101, segmentation module 102, description module 103, annotation module 104, and interaction module 105, to implement functions such as data annotation and obtain data annotation results.

[0064] I / O interface 112 is used to exchange data with external devices. A user can input data into I / O interface 112 via terminal device 130, such as an instruction to instruct computing device 110 to initiate execution of a data annotation method on target data. Alternatively, input data to I / O interface 112 can also come from database 120.

[0065] The preprocessing module 113 is used to preprocess the input data received by the I / O interface 112. In an embodiment of the present application, the preprocessing module 113 can be used to perform preprocessing operations such as fusion, alignment, and noise reduction on the input data received from the I / O interface 112 (e.g., fusion of videos captured by a binocular camera) to improve the accuracy of data annotation.

[0066] When the computing device 110 preprocesses input data, or when the computing module 111 of the computing device 110 performs calculations and other related processing, the computing device 110 can call data, codes, etc. in the data storage system 140 for corresponding processing, and can also store data and instructions obtained from the corresponding processing in the data storage system 140.

[0067] The data storage module 114 is used to convert the data marked by the calculation module 111 into a default or user-specified format, and then transmit it to the data storage system 140 for storage.

[0068] Finally, the I / O interface 112 returns the processing result to the terminal device 130, thereby providing it to the user so that the user can view the processing result. In an embodiment of the present application, the processing result can be a labeling result of the target data, such as multiple video segments and their corresponding first description information.

[0069] The terminal device 130 can be used as a data collection terminal to collect data such as Figure 2 The input data of the input I / O interface 112 and the processing result of the output I / O interface 112 are taken as new sample data and stored in the database 120. Of course, the sample data can also be collected without passing through the terminal device 130, but the I / O interface 112 can be used to collect the sample data. Figure 2 The input data of the input I / O interface 112 and the processing results of the output I / O interface 112 are stored in the database 120 as new sample data.

[0070] Figure 2 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. Figure 2 The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. According to the user's demand for data annotation, the data annotation system and computing equipment may include more or fewer hardware components, and this application does not limit this. Figure 2 In the embodiment, the data storage system 140 is an external memory relative to the computing device 110. In other cases, the data storage system 140 may also be placed in the computing device 110.

[0071] The application scenarios include, but are not limited to, automatic labeling of embodied intelligence data, displaying labeling results, and verifying labeled embodied intelligence data. The application can effectively and automatically label input target data, display the labeling results to the user, and support user verification of the labeling results. The application can also acquire target data from various devices, such as mobile hard drives and server base stations, and automatically label it.

[0072] Below Figure 2 The data annotation system 100 and the computing device 110 shown are based on and combined with Figure 3 The data annotation method provided in this application is described in detail, wherein: Figure 3 is a schematic diagram showing an interactive interface according to an exemplary embodiment. The interface can be displayed on Figure 2 The terminal device 130 in the display is displayed to the user. It should be understood that Figure 3The layout of each part, interactive controls and other contents are all examples. The display content of the interface can change according to changes in factors such as actual needs. The embodiments of the present application do not limit the specific display content of the interactive interface.

[0073] The data annotation method provided in this application can be divided into two stages, including: automatic annotation process and manual verification process. Figure 4 As shown, Figure 4 The figure is a flowchart of a data labeling method according to an exemplary embodiment.

[0074] In some possible embodiments, after receiving a trigger instruction to start the automatic annotation process, the computing device executes the data annotation method provided by this application on the target data, wherein the trigger instruction may be a user clicking on a button such as Figure 3 In the interactive interface shown, click the "Data Loading" control, select the data to be annotated, and then click the "Auto Annotation" control to instruct the computing device to start executing the data annotation method provided in this application on the target data. Alternatively, the trigger instruction can also be a set automation script that instructs the computing device to automatically obtain the target data from various devices such as mobile hard drives, server base stations, etc., and execute the data annotation method provided in this application on it. The embodiments of this application do not limit the implementation method of the trigger instruction.

[0075] The following first describes the automatic annotation process in detail.

[0076] Automatic annotation process

[0077] S201, a computing device obtains target data.

[0078] In this step, it can be Figure 2 The input module 101 in obtains target data, wherein the target data is embodied intelligence type data, such as the aforementioned rosbag data packet.

[0079] Specifically, the target data includes the motion data of the object at each moment in the process of the object performing the first task and the video of the object performing the first task. The target data can be obtained by the user by manipulating the robot to perform the first task, or by simulating the robot performing the first task in the simulation software. This application does not limit the source of the target data. Among them, the objects include humanoid robots, bionic robot dogs, or robotic arms with shooting devices deployed in real or simulated environments, etc. In some feasible embodiments, creatures such as people or dogs wearing relevant equipment (such as cameras and motion capture equipment) can also be objects in this application. In the embodiments of this application, the object is mainly described as a humanoid robot (referred to as robot).

[0080] The target data includes videos captured by cameras mounted on the robot or externally. The target data includes motion data, which refers to the motion and movement information of the robot while performing a task. This typically includes the robot's chassis speed and acceleration, the end-device's position, the opening and closing of its grippers, and the angles of its joints. This data can be captured by sensors mounted on the robot or, in some feasible embodiments, by motion capture equipment.

[0081] Exemplarily, the target data acquired by the computing device is generated by the user through remote control of the robot, and the target data is collected during the process of the robot performing the first task: "Please go to the refrigerator and get me a bottle of water". The robot is equipped with a binocular camera similar to the human eye. The user controls the robot to move over a larger range by remotely controlling the size and direction of the robot chassis speed, thereby performing tasks such as "navigating to the refrigerator". After the robot reaches the refrigerator, the user remotely controls the posture of the robot end and the opening and closing of the gripper, for example, by lifting the robot's mechanical arm, opening the gripper, and clamping the handle of the refrigerator door, thereby performing the task of "opening the refrigerator" and "taking out a bottle of water from the refrigerator". During the execution process, the robot's motion data at each moment and the video of the robot performing the first task are collected, and then a rosbag data packet containing the aforementioned video and motion data can be generated. When executing step S201, the computing device obtains the data packet.

[0082] S202: The computing device divides the video into multiple sub-videos according to the motion data.

[0083] In this step, it can be Figure 2 The segmentation module 102 in the embodiment divides the video into multiple sub-videos according to the motion data.

[0084] The video in the target data may be a video of an object performing a complex task. The purpose of step S202 is to split it into videos corresponding to multiple simple tasks. For example, a video corresponding to the object performing the complex task of "please go to the refrigerator and get me a bottle of water" is divided into sub-videos corresponding to the six simple tasks of "navigating to the refrigerator - opening the refrigerator - taking out water - closing the refrigerator - walking to the owner - handing over the water".

[0085] In the embodiment of the present application, a complex task similar to the aforementioned may also be referred to as a first task or a whole-segment task, and a simple task similar to the aforementioned may also be referred to as a second task, a subtask, or a segmented task.

[0086] Among them, at the starting moment and / or ending moment corresponding to each sub-video obtained by division, the action performed by the object meets the first specified feature; the first specified feature may refer to a specified change in the motion state of the object when the object performs the action according to the action data. For example, the robot is performing a task according to the action data. When the robot changes from performing subtask 1: navigating to the refrigerator to performing subtask 2: opening the refrigerator, the chassis speed of the robot will change from moving to stationary (so that the robot is stationary in front of the refrigerator). This change can be regarded as the specified change in the motion state of the object in the embodiment of the present application. Specifically, the first specified feature includes but is not limited to: the object changes from a moving state to a stationary state, from a stationary state to a moving state, and the occurrence of a turn in the action performed by the object.

[0087] In one possible implementation, the turning angle in the first specified feature needs to be greater than a set turning angle threshold (such as a threshold of 180 degrees). This can avoid dividing the video into too many sections and further improve the accuracy of video division.

[0088] For example, if the first task the object needs to perform is: "Please go to the refrigerator and get me a bottle of water", and the object is currently in front of a desk and is stationary, it can be understood that the object will then navigate to the refrigerator, and then the object will start to accelerate from the stationary state, and when it reaches the refrigerator, it will gradually decelerate until it stops. Obviously, there are two action data that meet the first specified characteristics in the action data of the time period corresponding to the above process, and then the video corresponding to "navigating to the refrigerator" can be captured through the moments corresponding to these two action data.

[0089] In one possible implementation, when the computing device divides a video into multiple sub-videos according to the action data, it can first determine at least one moment corresponding to the key action data in the action data that meets the first specified feature as at least one first segmentation moment. Then, the computing device determines every two adjacent first segmentation moments as the start moment and end moment of a sub-video, thereby obtaining multiple sub-videos.

[0090] Exemplarily, the computing device determines the segmentation moments including 14:28:00, 14:30:05, and 14:35:10 based on the key action data in the action data that meets the first specified characteristics, and then the computing device determines the first segmentation moments 14:28:00 and 14:30:05 as the starting moment and ending moment of a sub-video, and determines the first segmentation moments 14:30:05 and 14:35:10 as the starting moment and ending moment of another sub-video.

[0091] In order to further improve the accuracy of the computing device in dividing the video, the computing device can also determine the moment corresponding to the key video frame in the video that meets the second specified feature as the second segmentation moment, where the second specified feature can refer to a specified change in the video content of the video. The specific requirements of the specified change can be set according to actual needs and scenarios. For example, when a set target object appears in the video screen, it can be considered that the video content has undergone a specified change, etc., and then the computing device divides the video into multiple sub-videos according to the first segmentation moment and the second segmentation moment.

[0092] In one possible implementation, the computing device can compare the similarity of the video images frame by frame in chronological order (such as from early to late), and if the similarity between the two video frames is less than the set similarity threshold, it is determined that a specified change has occurred in the video content, and then the later moment corresponding to the two aforementioned video frames is determined as the second segmentation moment, and then the computing device divides the video into multiple sub-videos according to the first segmentation moment and the second segmentation moment.

[0093] Alternatively, in another possible implementation, the computing device may input the video into a video segmentation model, obtain multiple sub-videos output by the video segmentation model, and then determine the start and end times of each sub-video as the second segmentation time. The computing device then divides the video into multiple sub-videos based on the first segmentation time and the second segmentation time. The video segmentation model is a model capable of understanding video content and can segment the input video into multiple sub-videos based on changes in elements such as scenes and themes within the video content.

[0094] Through the above step S202, the computing device can accurately divide the video in the target data into multiple sub-videos. In other words, the computing device divides a longer video corresponding to a whole task into multiple shorter videos corresponding to each segmented task.

[0095] S203: The computing device obtains first description information of each sub-video.

[0096] In this step, it can be specifically through Figure 2 The description module 103 in the program describes each sub-video to obtain first description information of each sub-video, wherein the first description information is used to describe the second task performed by the object in the time period corresponding to each sub-video; the second task is a sub-task of the first task.

[0097] For example, if the first task is "Please go to the refrigerator and get me a bottle of water", the corresponding second task (subtask) may include one or more of "navigating to the refrigerator, opening the refrigerator, taking out the water, closing the refrigerator, walking to the owner, and delivering the water", and the first description information may be the description text of the second task, such as the aforementioned "open the refrigerator".

[0098] In one possible implementation, when obtaining the first description information of each sub-video, the computing device may input each sub-video into a video description model to obtain the first description information corresponding to each sub-video output by the video description model.

[0099] The video description model is a model capable of understanding video content. It can be a neural network model trained based on multiple video samples, their corresponding second description information samples, and first description information labels. It can output corresponding description information for changes in actions, scenes, and other aspects of the video content. Using this model, a computing device can improve the efficiency and accuracy of obtaining second description information.

[0100] Furthermore, in order to improve the accuracy of the first description information output by the video description model, the computing device may input the second description information and each sub-video into the video description model to obtain the first description information corresponding to each sub-video output by the video description model; wherein the second description information is used to describe the first task performed by the object.

[0101] For example, the video description model determines the first description as "Open the wardrobe" based on the video content. However, based on the second description input, "Please get me a bottle of water from the refrigerator," the model determines that the semantic distance between the two descriptions exceeds a set threshold. The model corrects the first description and outputs the following first description: "Open the refrigerator."

[0102] In this way, the video description model is provided with the description information of the first task (equivalent to the semantic center), which can further improve the accuracy of the first description information output by the video description model.

[0103] S204: The computing device outputs the labeled data.

[0104] In this step, it can be Figure 2 The annotation module 104 in the embodiment outputs annotation data based on the multiple sub-videos, the first description information, etc. obtained in the above steps, and when outputting the annotation data, it can be further divided into sample data and label data.

[0105] Specifically, the labeled data can be divided into two types: embodied task planning data (first labeled data) and embodied instruction execution data (second labeled data).

[0106] The following is a detailed description of the two types of data.

[0107] 1) Embodied task planning data.

[0108] In the embodied task planning data, specifically, its sample data includes: the second description information, each sub-video, and the label data includes: the first description information of each sub-video (description information of the subtask).

[0109] For example, a labeled embodied task planning data set might include: a video frame from a video corresponding to the first task description "Please go to the refrigerator and get me a bottle of water" and the subtask "Navigate to the refrigerator," with the following labels: Navigate to the refrigerator, open the refrigerator, take out the water, close the refrigerator, walk to the owner, and pass the water. Another labeled embodied task planning data set might include: a video frame from a video corresponding to the first task description "Please go to the refrigerator and get me a bottle of water" and the subtask "Take out the water," with the following labels: Take out the water, close the refrigerator, walk to the owner, and pass the water.

[0110] 2) Embodied instruction execution data.

[0111] Specifically, in the embodied instruction execution data, the sample data includes: each sub-video and the corresponding first descriptive information, and the label data includes: multiple sub-action data, where each sub-action data is obtained by dividing the action data according to at least one first segmentation moment. The first segmentation moment has been introduced above and will not be repeated here. For example, a labeled embodied task planning data is: the video corresponding to the subtask "open the refrigerator", the descriptive information "open the refrigerator" corresponding to the video, and the label is the action data with the same start and end time as the video. In other words, the label is the action data for the time period corresponding to the object executing the subtask "open the refrigerator".

[0112] Both embodied task planning data and embodied instruction execution data can be used to train the embodied intelligence model. Specifically, different parts of the embodied intelligence model can be trained separately. For example, embodied task planning data is used to train the embodied task planning model in the embodied intelligence model; embodied instruction execution data is used to train the embodied instruction execution model in the embodied intelligence model.

[0113] Through the above steps S201-S204, the computing device realizes automatic labeling of target data, thereby improving the labeling efficiency of embodied intelligent data.

[0114] In some feasible embodiments, the computing device may not display the annotated data, but may directly save the obtained annotated data. When saving the annotated data, it may be saved in a format set by the user, such as a JSON file. This application does not limit the storage format of the annotated data.

[0115] Manual verification process

[0116] The embodiment of the present application may also provide an interactive interface, wherein the interactive interface can be Figure 2 The terminal device 130 in the embodiment is displayed to the user. Figure 3 The interactive interface shown explains the manual verification process in detail.

[0117] S301, displaying an interactive interface.

[0118] Among them, the interactive interface can be used to display the labeled data. Figure 5 As shown, Figure 5 yes Figure 4 The illustrated embodiment shows a comparison diagram of an interactive interface before and after the automatic labeling process. The left side shows the process before the automatic labeling process, where it can be seen that each display area does not have specific data; the right side shows the process after the automatic labeling process, where each display area displays the corresponding data. For example, the segmented instruction display area displays the description information of segmented task 1: Navigate to the refrigerator, and the description information of segmented task 2: Open the refrigerator (there can be more segmented tasks, Figure 5 Only 2 are shown).

[0119] Figure 5 The interactive interface shown includes a variety of display areas or interactive controls, including: "Video display area", "Entire task", "Timeline", "Start", "End", "Previous frame of start frame", "Next frame of start frame", "Previous frame of end frame", "Next frame of end frame", "Segmented task setting area", "Add segmented data", "Automatic labeling", "Data saving", "Data loading", etc. The interactive interface shown in the figures in this application is only a possible example. The specific interface layout and controls can be determined according to actual needs, and this application does not impose any restrictions.

[0120] Taking the above-mentioned interactive controls as an example, the "whole task" and "segmented task setting area" interactive controls are used by users to manually input or modify the description information of each task or select from a drop-down box; the "add segmented data" interactive control is used by users to actively add segmented tasks; the "start", "end", "previous frame of start frame", "next frame of start frame", "previous frame of end frame", and "next frame of end frame" interactive controls are mainly used by users to manually modify the start and end times of the video; the "automatic labeling" and "data loading" interactive controls have been explained in the embodiments of this application and will not be repeated here; and the "data saving" interactive control will be explained below and will not be repeated here.

[0121] The manual verification process is mainly aimed at Figure 5The labeled data displayed on the right is manually verified through the various interactive controls in the interactive interface.

[0122] S302: If a triggering operation on an interactive control in the interactive interface is received, the computing device adjusts the annotation data according to a set function corresponding to the interactive control.

[0123] The setting function includes modifying the first description information and / or the second description information, and modifying the start time and / or the end time of each sub-video.

[0124] In this step, the user can first select the segmented task to be modified by clicking the interactive controls of different segmented tasks in the segmented task display area, as shown in the following example: Figure 6 and Figure 7 As shown, Figure 6 and Figure 7 Schematic diagrams showing the interactive interfaces of different segmented tasks.

[0125] Specifically, Figure 6 The user selects "Segmented Task 1: Navigate to the Refrigerator". Accordingly, the video display area can display any frame in the video corresponding to the segmented task, such as the first frame, and the "Start" and "End" on the timeline correspond to the start time and end time of the video corresponding to "Segmented Task 1: Navigate to the Refrigerator" respectively.

[0126] and Figure 7 The user selects "Segmented Task 2: Open the Refrigerator". Accordingly, the video display area can display any frame in the corresponding video of the segmented task, such as the first frame (the object has completed navigation to the refrigerator), and the "Start" and "End" on the timeline correspond to the start time and end time of the corresponding video of "Segmented Task 2: Open the Refrigerator" respectively.

[0127] Then, adjustments are made to a selected segmented task, for example, by dragging the "Start" and "End" interactive controls on the timeline to modify the start and end times of the video and action data corresponding to the segmented task.

[0128] Since the accuracy of manually dragging to adjust the start and end times of a video is limited, the interactive interface in this application also provides four interactive controls to improve accuracy, including "Previous frame of start frame", "Next frame of start frame", "Previous frame of end frame" and "Next frame of end frame". Taking the "Previous frame of start frame" interactive control as an example, the user can roughly determine the start time of the video by manually dragging the "Start" interactive control, and then click the "Previous frame of start frame" interactive control. The start time of the video will be adjusted to the previous frame of the current frame of the "Start" interactive control, and the "Start" interactive control will also move on the timeline accordingly. The functions of the other three interactive controls are similar and will not be repeated here.

[0129] After the user confirms that the labeled data meets the requirements, he can click Figure 7 The "Data Save" interactive control in the computing device can then save the annotated data in a format specified by the user, which can be Figure 2 The data saving module 114 in the storage module can save the annotated data into a user-specified format. Specifically, the annotated embodied intelligence data (including description information, video and action data) is converted into a user-specified data format (such as json or h5) and then stored in Figure 2 The data storage system 140 in FIG.

[0130] In some feasible embodiments, the user can choose to mark / save target data within a specified time period. The user can choose Figure 7 , and then click the "Save Data" interactive control. The computing device will generate and save the corresponding annotated data for the user in the format specified by the user. This improves the flexibility of data annotation and avoids excessive use of computing and storage resources.

[0131] Through the above steps S301-S302, the annotation data is intuitively displayed to the user through the client's interactive interface, and the annotation data is adjusted according to the triggering operations of the user and each interactive control, which can further improve the accuracy of the annotation data and meet user needs.

[0132] The above mainly introduces the scheme of the embodiment of the present application from the perspective of the method. It is understandable that, in order to realize the above functions, the data annotation device includes at least one of the hardware structure and software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present application.

[0133] The embodiment of the present application can divide the data annotation device into functional units according to the above method example. For example, each functional unit can be divided corresponding to each function, or two or more functions can be integrated into one processing unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0134] For example, Figure 8 The following is a schematic diagram of a data tagging apparatus 800 according to an exemplary embodiment of the present application. The data tagging apparatus 800 is applied to a computing device, or the data tagging apparatus 800 can be a computing device. The data tagging apparatus 800 includes:

[0135] The first acquisition module 810 is used to acquire target data; the target data includes action data and video at each moment during the process of the object performing the first task, the object performs actions according to the action data to perform the first task, and the video is a video of the object performing the first task.

[0136] The determination module 820 is used to determine at least one moment corresponding to at least one key action data in the action data as at least one first segmentation moment, and the action performed by the object according to the at least one key action data at the at least one first segmentation moment meets the first specified feature.

[0137] The division module 830 is used to divide the video into multiple sub-videos, wherein the at least one first segmentation moment is the start moment and / or end moment of each sub-video.

[0138] The second acquisition module 840 is used to obtain the first description information of each sub-video; the first description information is used to describe the second task performed by the object in the time period corresponding to each sub-video; the second task is a subtask of the first task.

[0139] For example, combined with Figure 4 , the first acquisition module 810 can be used to perform the following Figure 4 As shown in S201, the determination module 820 can be used to perform the following Figure 4 As shown in S202, the partitioning module 830 can be used to perform the following steps: Figure 4 As shown in S203, the second acquisition module 840 can be used as follows Figure 4 S204 shown.

[0140] In one possible implementation, the data labeling device also includes an output module, which is used to: output labeling data, the labeling data including the multiple sub-videos and the first description information of each sub-video, the labeling data is used to train the embodied intelligent model, the object performs the action corresponding to the task through the embodied intelligent model, and the labeling data includes sample data and label data.

[0141] In one possible implementation, if the annotation data is first annotation data, the first annotation data also includes second description information, the second description information is used to describe the first task, each sub-video and the second description information are the sample data in the first annotation data, and the first description information of each sub-video is the label data.

[0142] In one possible implementation, if the annotation data is second annotation data, the second annotation data also includes multiple segments of sub-action data, each sub-video and the corresponding first description information are sample data in the second annotation data, and the multiple segments of sub-action data are determined as label data in the annotation data; the multiple segments of sub-action data are obtained by dividing the action data according to the at least one first segmentation moment, and the at least one first segmentation moment is the starting moment and / or ending moment of each segment of sub-action data.

[0143] In one possible implementation, the division module 830 is further used to: determine at least one moment corresponding to at least one key video frame in the video as at least one second segmentation moment; the video content of the video meets the second specified feature at the at least one second segmentation moment; divide the video into multiple sub-videos, wherein the at least one first segmentation moment or the at least one second segmentation moment is the starting moment and / or ending moment of each sub-video.

[0144] In a possible implementation, the data tagging device further includes an interaction module, the interaction module being configured to: provide an interaction interface, the interaction interface being configured to display each sub-video and the respective first description information of each sub-video;

[0145] Receive a triggering operation on an interactive control in the interactive interface, and adjust the annotation data according to a setting function corresponding to the interactive control; the setting function includes modifying the first description information, and / or the second description information, the start time, and / or the end time of each sub-video.

[0146] As a feasible example, the data labeling device 800 provided in this application is implemented through a software module. For example, the software module can be provided to users through a cloud service subscription model, and users can choose different subscription levels according to their needs; for example, the software module can also provide enterprise-level customized services with professional domain customization, interface personalization and extended functions according to the needs of users or enterprises.

[0147] In addition, the data annotation device 800 provided in this application can also be made into a value-added service and provided to users, which is not limited in this application. When the data annotation device 800 is implemented as a software module, the data annotation device 800 can also be embedded in other model training software or data annotation tool chain systems.

[0148] The present application embodiment also provides a computing device 100. Figure 9 As shown, computing device 100 includes: bus 102, processor 104, memory 106, and communication interface 108. Processor 104, memory 106, and communication interface 108 communicate with each other via bus 102. Computing device 100 can be a server or a terminal device. It should be understood that the embodiments of the present application do not limit the number of processors and memories in computing device 100.

[0149] The bus 102 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 9 The bus 104 may include a path for transmitting information between various components of the computing device 100 (eg, memory 106, processor 104, communication interface 108).

[0150] The processor 104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0151] The memory 106 may include volatile memory, such as random access memory (RAM). The processor 104 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0152] The memory 106 stores executable program code, and the processor 104 executes the executable program code to implement the functions of the first acquisition module, the determination module, the division module, and the second acquisition module, thereby implementing the data labeling method. In other words, the memory 106 stores instructions for executing the data labeling method.

[0153] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0154] like Figure 10 As shown, the computing device cluster includes at least one computing device 100. The memory 106 in one or more computing devices 100 in the computing device cluster may store the same instructions for executing the data labeling method.

[0155] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store some instructions for executing the data labeling method. In other words, the combination of one or more computing devices 100 can jointly execute the instructions for executing the data labeling method.

[0156] It should be noted that the memory 106 in different computing devices 100 in the computing device cluster can store different instructions, each for executing part of the functions of the data labeling apparatus. In other words, the instructions stored in the memory 106 in different computing devices 100 can implement the functions of one or more of the first acquisition module, the determination module, the partitioning module, and the second acquisition module.

[0157] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network, which may be a wide area network or a local area network. Figure 11 A possible implementation is shown. Figure 11 As shown, two computing devices 100A and 100B are connected via a network. Specifically, the connection to the network is achieved through a communication interface within each computing device. In this possible implementation, the memory 106 within computing device 100A stores instructions for executing the functions of the first acquisition module. Simultaneously, the memory 106 within computing device 100B stores instructions for executing the functions of the determination module, the partitioning module, and the second acquisition module.

[0158] It should be understood that Figure 11 The functions of the computing device 100A shown in FIG. 1 may also be completed by multiple computing devices 100. Similarly, the functions of the computing device 100B may also be completed by multiple computing devices 100.

[0159] The present application also provides a computer program product comprising instructions. The computer program product may be software or a program product comprising instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the data labeling method.

[0160] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the data labeling method, or instruct the computing device to execute the data labeling method.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A data annotation method, characterized in that: The method comprises: Acquire target data; the target data includes action data and video at various moments during the process of an object performing a first task, the object performing actions according to the action data to perform the first task, and the video is a video of the object performing the first task; determining at least one moment corresponding to at least one key action data in the action data as at least one first segmentation moment, wherein the action performed by the object according to the at least one key action data at the at least one first segmentation moment meets a first specified feature; Dividing the video into multiple sub-videos, wherein the at least one first segmentation moment is a start moment and / or an end moment of each sub-video; Obtain first description information of each sub-video; the first description information is used to describe a second task performed by the object in a time period corresponding to each sub-video; the second task is a subtask of the first task.

2. The method according to claim 1, characterized in that The method further comprises: Output labeled data, where the labeled data includes the multiple sub-videos and the first description information of each sub-video. The labeled data is used to train an embodied intelligent model, and the object performs actions corresponding to the task through the embodied intelligent model. The labeled data includes sample data and label data.

3. The method according to claim 2, characterized in that If the labeled data is the first labeled data, the first labeled data also includes second description information, the second description information is used to describe the first task, each sub-video and the second description information are the sample data in the first labeled data, and the first description information of each sub-video is the label data.

4. The method according to claim 2, characterized in that If the annotation data is the second annotation data, the second annotation data also includes multiple segments of sub-action data, each sub-video and the corresponding first description information are sample data in the second annotation data, and the multiple segments of sub-action data are determined as label data in the annotation data; the multiple segments of sub-action data are obtained by dividing the action data according to the at least one first segmentation moment, and the at least one first segmentation moment is the starting moment and / or ending moment of each segment of sub-action data.

5. The method according to any one of claims 1 to 4, characterized in that The step of dividing the video into a plurality of sub-videos includes: determining at least one moment corresponding to at least one key video frame in the video as at least one second segmentation moment; wherein the video content of the video meets a second specified feature at the at least one second segmentation moment; The video is divided into multiple sub-videos, wherein the at least one first segmentation moment or the at least one second segmentation moment is the start moment and / or end moment of each sub-video.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Providing an interactive interface, the interactive interface being used to display each of the sub-videos and respective first description information of each of the sub-videos; Receive a triggering operation on an interactive control in the interactive interface, and adjust the annotation data according to a setting function corresponding to the interactive control; the setting function includes modifying the first description information, and / or the second description information, the start time, and / or the end time of each sub-video.

7. A data labeling device, characterized in that: The device comprises: A first acquisition module is configured to acquire target data; the target data includes action data and a video of an object at various moments in the process of performing a first task, the object performing actions according to the action data to perform the first task, and the video is a video of the object performing the first task; a determining module, configured to determine at least one moment corresponding to at least one key action data in the action data as at least one first segmentation moment, wherein the action performed by the object according to the at least one key action data at the at least one first segmentation moment meets a first specified characteristic; a division module, configured to divide the video into a plurality of sub-videos, wherein the at least one first division moment is a start moment and / or an end moment of each sub-video; The second acquisition module is used to obtain the first description information of each sub-video; the first description information is used to describe the second task performed by the object in the time period corresponding to each sub-video; the second task is a subtask of the first task.

8. The data labeling device according to claim 7, characterized in that: The data labeling device further includes an output module, which is used to: Output labeled data, where the labeled data includes the multiple sub-videos and the first description information of each sub-video. The labeled data is used to train an embodied intelligent model, and the object performs actions corresponding to the task through the embodied intelligent model. The labeled data includes sample data and label data.

9. The data labeling device according to claim 8, characterized in that: If the labeled data is the first labeled data, the first labeled data also includes second description information, the second description information is used to describe the first task, each sub-video and the second description information are the sample data in the first labeled data, and the first description information of each sub-video is the label data.

10. The data labeling device according to claim 8, characterized in that: If the annotation data is the second annotation data, the annotation data also includes multiple segments of sub-action data, each sub-video and the corresponding first description information are sample data in the second annotation data, and the multiple segments of sub-action data are determined as label data in the annotation data; the multiple segments of sub-action data are obtained by dividing the action data according to the at least one first segmentation moment, and the at least one first segmentation moment is the starting moment and / or ending moment of each segment of sub-action data.

11. The data labeling device according to any one of claims 7 to 10, characterized in that: The partitioning module is used for: determining at least one moment corresponding to at least one key video frame in the video as at least one second segmentation moment; wherein the video content of the video meets a second specified feature at the at least one second segmentation moment; The video is divided into multiple sub-videos, wherein the at least one first segmentation moment or the at least one second segmentation moment is the start moment and / or end moment of each sub-video.

12. The data labeling device according to any one of claims 7 to 11, characterized in that: The data labeling device further includes an interaction module, which is configured to: Providing an interactive interface, the interactive interface being used to display each of the sub-videos and respective first description information of each of the sub-videos; Receive a triggering operation on an interactive control in the interactive interface, and adjust the annotation data according to a setting function corresponding to the interactive control; the setting function includes modifying the first description information, and / or the second description information, the start time, and / or the end time of each sub-video.

13. A computing device cluster, characterized in that: comprising at least one computing device, each computing device including a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in a memory of the at least one computing device, so that the computing device cluster executes the data labeling method according to any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium includes computer instructions; when the computer instructions are executed in a computing device, the computing device executes the data labeling method according to any one of claims 1 to 6.

15. A computer program product, characterized in that When the computer program product is run in a computing device, the computing device executes the data labeling method according to any one of claims 1 to 6.

Images