Annotation device, annotation method, and annotation program

Abstract

To reduce time and cost in construction of robot data with a language label for machine learning by automatically imparting the language label to robot time series data for each subtask data.SOLUTION: The annotation device 1 includes the task decomposition unit 30 that decomposes the robot time-series data related to the execution of the task into the plurality of subtask data according to the language instruction including the designation of the task input in the natural language, generates the language label indicating the content of the subtask data for each of the plurality of subtask data, and generates the subtask file in which the language label is given to the subtask data, and the storage unit 40 that stores the subtask file.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an annotation device, an annotation method, and an annotation program. [Background technology]

[0002] In recent years, for example in the field of robotics, research has been conducted on using machine learning models to have robots perform tasks corresponding to abstract instructions (linguistic instructions) in human natural language. Such research uses, for example, video data of a robot successfully performing a task, annotation data such as labels attached to the video data, and a machine learning model to train the robot's control for that task. Using such a machine learning model, it is possible to infer the currently observed state of the robot's environment and the robot's control for executing human linguistic instructions.

[0003] Patent Documents 1 and 2 disclose video captioning software technology that adds text explanations to videos. Patent Document 3 discloses robot motion editing software technology that graphically edits robot motion. These technologies add annotation data to videos. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-073709 [Patent Document 2] Japanese Patent Application Publication No. 2019-087116 [Patent Document 3] Japanese Patent Publication No. 2024-021611

[0005] When applying machine learning to robotics, dividing tasks into smaller pieces and labeling the subtasks contributes greatly to improving model performance. Even if the overall set of tasks is different, they often contain common small tasks (subtasks). For example, the tasks "moving fruit" and "assembling a machine" have different overall objectives, but both require the subtask of "grasping." The performance of a machine learning model depends heavily on the diversity and quantity of training data. Therefore, from the perspective of data augmentation, it is important to divide and label data for each subtask, not just the entire task.

[0006] However, manually assigning labels to each subtask requires a significant amount of time and cost. For example, a task involving assembling a product may take several tens of minutes, but subtasks such as "grabbing" can be completed within a few seconds, resulting in hundreds of subtasks. Therefore, it is a significant burden for a human to accurately observe and label the video of scenes corresponding to all of these subtasks. Summary of the Invention [Problem to be solved by the invention]

[0007] The present disclosure provides an annotation device that can reduce the burden of annotation. [Means for solving the problem]

[0008] The present disclosure provides an annotation device including: a task decomposition unit that, in response to a linguistic instruction including a task specification input in natural language, decomposes robot time-series data regarding the execution of the task into a plurality of subtask data, generates a linguistic label for each of the plurality of subtask data indicating the content of the subtask data, and generates a subtask file in which the linguistic label is assigned to the subtask data; and a storage unit that stores the subtask file.

[0009] The present disclosure provides an annotation method that, in response to a linguistic instruction including a task specification input in natural language, decomposes robot time-series data related to the execution of the task into a plurality of subtask data, generates a linguistic label for each of the plurality of subtask data indicating the content of the subtask data, generates a subtask file in which the linguistic label is assigned to the subtask data, and stores the subtask file.

[0010] The present disclosure provides an annotation program that causes a computer to execute the following steps in response to linguistic instructions including a task specification input in natural language: decomposing robot time-series data related to the execution of the task into multiple subtask data; generating, for each of the multiple subtask data, a linguistic label indicating the content of the subtask data; generating a subtask file in which the linguistic label is assigned to the subtask data; and storing the subtask file. [Effects of the Invention]

[0011] According to the present disclosure, robot time-series data is decomposed into subtask data, and language labels are generated and assigned to the subtask data, so that language labels can be automatically assigned to each subtask of the robot time-series data, thereby reducing the time and cost required to build language-labeled robot data for machine learning. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a conceptual diagram illustrating an annotation device according to an embodiment of the present disclosure and an image of annotation performed by the annotation device. [Figure 2] FIG. 1 is a block diagram illustrating a configuration example of an annotation device according to an embodiment of the present disclosure. [Figure 3] A diagram showing a list of subtask data obtained by decomposing robot time-series data using a task decomposition model. [Figure 4] Diagram showing subtask data displayed in a hierarchical structure [Figure 5] The list of subtask data in tree format that is actually displayed in the subtask display area [Figure 6] FIG. 10 is an explanatory diagram showing details of processing by a subtask recommendation unit. [Figure 7] Flowchart showing the procedure of the annotation method executed by the annotation device (part 1) [Figure 8] Flowchart showing the procedure of the annotation method executed by the annotation device (part 2) DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, with reference to the accompanying drawings as appropriate, detailed descriptions of each embodiment specifically disclosing an annotation device, annotation method, and annotation program according to an embodiment of the present disclosure will be provided. However, more detailed descriptions than necessary may be omitted. For example, detailed descriptions of well-known matters or redundant descriptions of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter recited in the claims.

[0014] 1 is a conceptual diagram illustrating an annotation device according to an embodiment of the present disclosure and an image of annotation performed by the annotation device. The annotation device of this embodiment is a device for realizing annotation of robot data for a robot to perform machine learning. In particular, the annotation device of this embodiment automatically decomposes a task to be executed into subtasks and can draw them on a GUI (Graphical User Interface), thereby enabling efficient annotation and creation of learning data.

[0015] The annotation device of this embodiment can handle multimodal robot data, i.e., multiple data such as video, audio, force, and tactile data. As an example, a user inputs a linguistic instruction (natural language) such as "Put the apple in the box" in some form (text, voice, etc.) to a robot that grabs an apple and puts it in a box. The annotation device uses natural language processing technology or a machine learning model, such as a Vision Language Model (VLM) that also takes images into account, to decompose the task to be executed (putting the apple in the box) into subtasks. The annotation device records data for each subtask (subtask data, described below) in a tree-like data structure, and a GUI displays linguistic labels representing the subtasks in a tree format. When decomposing a task into subtasks, the annotation device can improve the confidence of the subtasks by referring to domain knowledge. By referencing a subtask file with automatically assigned linguistic labels, a user can easily understand the list of subtasks and edit them as needed, thereby reducing the time and cost required to build linguistically labeled robot data.

[0016] 2 is a block diagram showing an example configuration of an annotation device according to an embodiment of the present disclosure. The annotation device 1 includes an input device 10, a database (DB) 20, a task decomposition unit 30, a storage unit 40, a GUI 50, and a subtask recommendation unit 60. The annotation device 1 may be realized by a single computer, or each element may be a separate computer connected to each other via a network.

[0017] The input device 10 is a device through which a user inputs various information in the form of text, voice, etc., and includes a keyboard, mouse, touch panel, microphone, etc. The input device 10 is equipped with an operation device 11 and a language instruction unit 12. The operation device 11 is a device for inputting information for operating the annotation device 1. The language instruction unit 12 is a device through which a user inputs instructions in natural language to cause a robot targeted by the annotation device 1 to perform a specific task. The input may be in the form of text or voice. The language instruction is an instruction that a user inputs to a robot, an instruction that includes the specification of a task, and an instruction in a natural language that is normally used by humans.

[0018] The DB 20 stores data necessary for the annotation device 1 to execute a task according to a linguistic instruction input by a user from the input device 10. The DB 20 stores domain knowledge 21 and robot data 22.

[0019] The domain knowledge 21 is, for example, knowledge data related to the type of robot and the content of the task. The domain knowledge 21 is used to generate candidates for subtask data generated by the task decomposition unit 30 (described later) and to calculate confidence levels. The domain knowledge 21 is stored in the DB 20 in text format, for example, as prior knowledge for inference.

[0020] Examples of domain knowledge include the following:

[0021] 1. Object characteristics and attributes - Shape, size, color, etc. of apples, boxes, and work tables 2. Object position and initial state Initial position of the apple (e.g. inside the apple basket) - Initial location of the box (e.g. on the workbench) Initial position and posture of the robot 3. Task Objectives and Requirements Objective: Packing apples into boxes Requirements: Putting apples into the box without damaging them, number of apples to fill the box, etc. 4. Task Procedures and Constraints Optimal packing method for apples (e.g. packing efficiently without gaps) Box capacity limit (e.g. maximum number of apples that can fit in a box) Time constraints (e.g., the task must be completed within a certain time frame) 5.Robot Functions and Limitations The robot's grasping ability (e.g., the type of gripper that can grab an apple) -Robot range of motion and degrees of freedom Sensor type and accuracy (e.g. camera, force sensor, etc.)

[0022] The robot data 22 is at least one of video, audio, force, and tactile data, including video of the robot performing a task, information such as coordinates of sensed objects, and physical information such as joint angles of the robot. In other words, the robot data may be multimodal data.

[0023] The task decomposition unit 30 decomposes the robot time-series data related to the execution of a task into a plurality of subtasks, specifically into subtask data, in response to a linguistic instruction including a task specification input in natural language. Furthermore, the task decomposition unit 30 generates a linguistic label indicating the content of each of the plurality of subtask data, and generates a subtask file 41 in which the generated linguistic label is assigned to the subtask data.

[0024] Subtask data is more detailed unit of behavioral data required to execute language instructions. Subtask data includes not only leaf nodes but also intermediate nodes. A subtask file is a file that stores subtask data information (hierarchical structure, labels, start and end times, etc.) and is output in a format such as JSON (JavaScript Object Notation).

[0025] That is, the task decomposition unit 30 selects robot time-series data representing the results of controlling the robot from the robot data 22 in response to a task related to a linguistic instruction input by the user from the input device 10, and automatically decomposes the data into subtask data using natural language processing technology and a machine learning model to generate a subtask file 41. The task decomposition unit 30 uses a combination of natural language processing technology and a machine learning model as a task decomposition model, and generates subtask data candidates by utilizing the domain knowledge 21 and the robot data 22.

[0026] The task decomposition model is a model that uses natural language processing technology or a machine learning model and is used to decompose linguistic instructions into subtask data. In the embodiment, the task decomposition model is assumed to be a model with advanced inference capabilities, such as a large language model (LLM). In particular, the embodiment processes robot time-series data that may include images, so it uses a model such as a VLM that can handle images and text in combination.

[0027] LLMs and VLMs generally learn a variety of text data and image data regardless of the technical field so that they can be used for general-purpose applications. Therefore, if an LLM or VLM is used as is to decompose linguistic instructions into subtask data, it is not necessarily possible to obtain subtask data suitable for the current task or robot control. Therefore, in this embodiment, domain knowledge 21 is used to allow the task decomposition model to learn the current task and robot constraints. An example of using domain knowledge to improve the performance of planning using VLMs and LLMs is the paper "Vision-Language Interpreter for Robot Task Planning" (2024 IEEE International Conference on Robotics and Automation, February 20, 2024). Therefore, in this embodiment, details of a method for improving the performance of VLMs and LLMs using domain knowledge are omitted.

[0028] Specifically, when a linguistic instruction is input from the user, the task decomposition unit 30 acquires domain knowledge 21 and robot data 22 from the DB 20. The task decomposition unit 30 analyzes the linguistic instruction, domain knowledge 21, and robot data 22 using natural language processing technology or a machine learning model, estimates robot time-series data corresponding to the task structure, and then generates a list of subtask data. The subtask data is linguistically hierarchized in a format such as JSON, and the task decomposition unit 30 outputs a subtask file 41 in which the timestamps of the associated robot data are stored.

[0029] The storage unit 40 is a storage device that stores the subtask file 41 generated by the task decomposition unit 30. The storage unit 40 may store a program (annotation program) for executing the annotation method executed by the annotation device 1 as shown in Figures 7 and 8. The computers that constitute each part of the annotation device 1 can read this program and execute each procedure.

[0030] The GUI 50 is a device for visually and editably displaying information and data to be processed by the user. The GUI 50 includes a subtask display unit 51, a data input unit 52, a data control unit 53, and a screen display unit 54.

[0031] The subtask display unit 51 functions as a display control unit that displays in an editable manner at least the language labels of the subtasks generated by the task decomposition unit 30. In other words, the subtask display unit 51 displays the structure of the subtask data stored in the subtask file 41 in a format that can be edited by the user.

[0032] The data input unit 52 functions as an input unit that accepts edits to at least the language labels of the subtask data displayed by the subtask display unit 51 through the user's operation of the operation device 11, and updates the subtask file 41. The data input unit 52 accepts edits to the subtasks made by the user, and updates the subtask file 41. The user uses the operation device 11 to input edits to at least the subtask names (language labels), and the data input unit 52 updates the subtask file 41 in accordance with this input. The user can also input edits to the data input unit 52, such as the start time and end time of the subtask data.

[0033] When an update is made from data input unit 52 , subtask display unit 51 reflects the updated contents from data input unit 52 in subtask file 41 and performs display based on updated subtask file 41 .

[0034] The data control unit 53 controls the playback of robot time-series data in response to a user's operation of the operation device 11. Specifically, the data control unit 53 accepts operations such as playback, stop, time specification, and subtask tree selection of the robot time-series data, and reflects these operations in the image display.

[0035] The screen display unit 54 is a display device (display) for displaying images. The screen display unit 54 can display subtask data controlled by the subtask display unit 51, updates from the data input unit 52, playback data of robot time-series data controlled by the data control unit 53, etc. in a manner that allows the user to visually confirm them.

[0036] FIG. 3 is a diagram showing a list of subtask data obtained by the task decomposition unit 30 decomposing a predetermined task using a task decomposition model. When the user inputs the following first linguistic instruction (prompt) from the linguistic instruction unit 12 of the input device 10, the task decomposition unit 30 decomposes the robot time-series data related to the execution of the task into subtask data and generates the list shown in FIG. 3. The screen display unit 54 may display this list. The subtask data is assigned not only a linguistic label but also a timestamp including the start time and end time.

[0037] (First Language Instruction) For the following task, please list detailed subtasks in chronological order that can be discerned from relevant multimodal data (e.g., video, audio, haptic). Subtasks should include start and end times. Please break down subtasks into as many granularities as possible, but ensure they are the smallest meaningful unit of action. Task: "Fill the box with apples" Domain Knowledge: (import text from a domain knowledge base)

[0038] When the subtask display unit 51 generates the list of Figure 3, the user actually confirms that the screen display unit 54 has displayed this list, and then inputs the following second language instruction from the language instruction unit 12 of the input device 10. The task decomposition unit 30 stores a hierarchical structure in which related subtask data is grouped in the subtask file 41, and the subtask display unit 51 displays the hierarchical structure of the subtasks. Figure 4 is a diagram of the subtask data displayed in a hierarchical structure.

[0039] (Second Language Instruction) Using the subtask list below, group semantically related subtasks and generate a hierarchical subtask tree. Express each node as a JSON object that includes the task name, start time, end time, and a list of subtasks (child nodes). For parent nodes that group multiple subtasks, set the start and end times so that they encompass the time range of the child nodes. Determine the tree hierarchy based on the abstraction level of the tasks. Place more abstract tasks at higher nodes and more specific subtasks at lower nodes. Consider the semantic relevance of the subtasks and group them at an appropriate level of granularity. Output in JSON format. Subtask List: 1. 00:00:00 - 00:00:05 Move your right hand closer to the box 2. ~~ 3. 00:01:35 - 00:01:40 Close the box

[0040] The hierarchical structure in Figure 4 is a data structure for expressing the hierarchical relationship of subtask data. Parent nodes represent general tasks, and child nodes represent detailed subtasks. The user can set the number of levels and the maximum number of child nodes per node, or the system will automatically determine this if no settings are made.

[0041] In the above description, in the flow of Figures 3 and 4, the user issues two-stage linguistic instructions (two-stage prompt input) including first and second linguistic instructions. The first linguistic instruction corresponds to the generation of a list of subtask data, and the second linguistic instruction corresponds to the generation of a list of a hierarchical structure with timestamps. However, the user does not necessarily need to issue two-stage linguistic instructions; the task decomposition unit 30 can perform two-stage processing even with only a single linguistic input that combines the two stages of linguistic instructions. Furthermore, the examples of the first and second linguistic instructions are merely examples, and linguistic instructions may be expressed in other ways depending on the development of the language model used and prompt engineering.

[0042] In the annotation device 1 according to this embodiment, the task decomposition unit 30 decomposes robot time-series data related to task execution into subtask data and generates a subtask file 41 in which a language label is assigned to each piece of subtask data, and the storage unit 40 can store the subtask file 41. As a result, the task decomposition unit 30 decomposes the robot time-series data into subtask data, generates language labels, and assigns them to the subtask data, so that language labels can be automatically assigned to each piece of subtask data in the robot time-series data. This reduces the time and cost required to build language-labeled robot data for machine learning.

[0043] Furthermore, the subtask display unit 51, which serves as a display control unit, displays at least the language labels of the subtask data in an editable manner, and when the data input unit 52, which serves as an input unit, accepts the edited content of the language labels and updates the subtask file 41, the subtask display unit 51 can display data based on the updated subtask file 41. This allows the assigned language labels to be displayed in an editable manner, enabling users to edit the language labels. This significantly reduces time and costs compared to when language labels are assigned manually from scratch.

[0044] The task decomposition unit 30 can also store a hierarchical structure in which related subtask data is grouped in the subtask file 41. The hierarchical structure shown in FIG. 4 is shown in JSON format. The subtask display unit 51 displays a list of subtask data in a tree-like hierarchical structure that is easily recognizable by humans, as shown in FIG. 5 (or the "task tree" on the right side of FIG. 1), on the screen display unit 54, assuming that in an actual work site, a user will visually view and edit the subtask data (start and end times are omitted). In other words, the hierarchical structure in FIG. 4 corresponds to part of the program for displaying the task tree on the right side of FIG. 1 and the tree structure in FIG. 5.

[0045] This allows the hierarchical structure of the subtask data to be displayed in a tree format, making it easier to intuitively grasp related subtask data and to easily check and edit language labels.

[0046] Furthermore, in this embodiment, the data control unit 53 can control the playback of robot time-series data in response to user operations. Specifically, the data control unit 53 accepts operations such as playback, stop, time specification, and subtask tree selection, and reflects these operations in the image display on the screen display unit 54. This allows the user to edit language labels while playing back the robot time-series data, thereby enabling efficient confirmation and editing of language labels.

[0047] The robot time-series data may be multimodal sensor data including at least one of video, audio, force, and tactile data, which allows tasks related to at least one of video, audio, force, and tactile data to be handled.

[0048] As described above, the task decomposition unit 30 can generate subtask data candidates by using a task decomposition model that combines natural language processing technology and a machine learning model and by referencing a knowledge base according to the type of robot or domain knowledge, i.e., the domain of the task. This makes it possible to generate appropriate subtask data candidates.

[0049] The data input unit 52 can also accept settings of the start time and end time of a subtask from the operation device 11 of the input device 10 by the user (editing the start time and end time), and update the subtask file in accordance with the settings. In this case, the subtask display unit 51 can perform display based on the updated subtask file 41 on the screen display unit 54 in conjunction with the playback of the robot time-series data by the data control unit 53. This allows the updated subtask file 41 to be displayed when the robot time-series data is played back.

[0050] The annotation device 1 further includes a subtask recommendation unit 60. The subtask recommendation unit 60 estimates a confidence level indicating the feasibility of each subtask data item from the robot time-series data and task information, and changes the presentation mode of the subtask data item depending on the confidence level. For example, the annotation device 1 preferentially presents the subtask data item to the user depending on whether the confidence level exceeds a predetermined threshold.

[0051] The confidence level is a value indicating the feasibility of each subtask data, i.e., the likelihood that the robot can actually execute the subtask corresponding to each subtask data. In the subtask recommendation unit 60, the confidence level is output by a confidence level estimation model 61 using robot data and linguistic information as input information, as shown in FIG. 6 . Here, the linguistic information is information containing natural language, including the task specified by the user and the generated subtask data. The confidence level increases, for example, as the similarity between the robot data and the linguistic information increases. Since the robot data is, for example, video data showing successful examples of a robot executing a task, similarity between the robot data and the linguistic information indicates a high probability that the subtask data can actually be executed by the robot. The confidence level decreases, for example, when task decomposition is poor or when the decomposed subtasks do not match the environmental information. The input information may also include information about the robot's state and the environment. When the user sets new subtask data, the confidence level of the entire task is recalculated to reflect the impact of modifying the linguistic labels of the subtask data on each node. The confidence estimation model 61 is configured using a base model such as a virtual machine learning model (VLM). The VLM has a structure for mapping images and natural language to a feature space and evaluating similarity. Therefore, it can estimate the similarity in the feature space between the images, etc., included in the input information and the language labels of the subtasks. This similarity does not include time-series information because it is the similarity between the images, etc., and the language labels, etc., but it can be used to calculate confidence. For example, the VLM estimates the similarity between each of multiple frames included in a series of videos showing the robot actually performing an action and the corresponding language labels, etc., and calculates statistical values ​​such as the mean and variance of the estimated similarity. This value represents the similarity between the video of the robot actually performing the action and the corresponding language labels, etc., and can be used as confidence. Alternatively, for example, the hand position and orientation in a frame a predetermined time later can be estimated from the hand position and orientation in a given frame and information such as the hand's range of motion and movement speed, and the similarity indicating the degree of similarity to the desired position and orientation corresponding to the language label can be used as confidence.Furthermore, the confidence level may be determined by combining the similarity for each frame and the similarity that takes into account the time-series factors between frames. As described above in the task decomposition model, VLM does not necessarily provide an evaluation appropriate for the current task or robot control. Therefore, domain knowledge 21 may be used to obtain more appropriate evaluation results from the confidence level estimation model 61. A model other than VLM may also be used as the confidence level estimation model 61. For example, the structure of the subtask tree may be expressed in text, and then LLM may be used to evaluate the validity of the correspondence between the subtask and the language label at each node. The confidence level estimation model 61 may also be a model other than the base model, such as a model trained as a value function.

[0052] The predetermined threshold is a reference value for determining whether to preferentially present a subtask to the user based on the certainty of the subtask. That is, the subtask recommendation unit 60 selects a subtask to preferentially present to the user based on the certainty and the threshold. Thus, the subtask recommendation unit 60 can preferentially present specific subtask data according to its feasibility.

[0053] Once the subtask recommendation unit 60 estimates the confidence level for each subtask, the subtask display unit 51 displays the subtask data on the screen display unit 54 with a priority level according to the confidence level. The display of the subtask data with a priority level according to the confidence level is not limited to a specific form. For example, the subtask display unit 51 may change the display color of the subtask data depending on the level of the confidence level. For example, to alert the user, subtasks with low confidence levels may be colored red or the like and displayed on the screen display unit 54.

[0054] As a specific example, suppose the confidence level for the subtask data with the subtask name "grab an apple" is estimated to be 0.73, and the confidence level for the subtask data with the subtask name "move your hand to the open box" is estimated to be 0.2. In this case, the subtask recommendation unit 60 causes the subtask display unit 51 to display the subtask data with a low confidence level, "move your hand to the open box," in color (in bold in FIGS. 5 and 6), as shown in FIGS. 5 and 6, and present it to the user preferentially. This allows the user to focus on the subtask data with a low confidence level and determine preferentially whether or not to edit the subtask name. However, such a function of preferentially presenting subtask data by the subtask recommendation unit 60 is not essential.

[0055] The subtask display unit 51 can also expand or collapse the tree and display it in response to a user's operation on the operation device 11 of the input device 10. This allows the hierarchical tree structure to be changed and displayed according to the user's wishes. For example, when the user uses the mouse of the operation device 11 to click on the mark M representing the subtask data of the parent node ("grab the apple") in FIG. 5, the subtask data (child nodes) below it may be expanded and displayed.

[0056] 7 and 8 are flowcharts showing the procedure of the annotation method executed by the annotation device 1. When a user selects, from the operation device 11 of the input device 10, robot data 22 related to a task that the user wants to use as the target of learning data, the data control unit 53 renders a moving image (video data) included in the robot time-series data on the screen display unit 54 (step S1).

[0057] The task decomposition unit 30 determines whether or not a linguistic instruction for the task (corresponding to the first linguistic instruction described above) has been input by the user from the linguistic instruction unit 12 of the input device 10 (step S2). If a linguistic instruction for the task has been input (Yes in step S2), the task decomposition unit 30 accepts the linguistic instruction for the task (step S3) and acquires the domain knowledge 21 and the robot data 22 from the DB 20 (step S4).

[0058] In the processing of step S5, the task decomposition unit 30 uses a combination of natural language processing technology and a machine learning model as a task decomposition model, and decomposes the robot time-series data selected from the robot data 22 corresponding to the task into subtask data by referring to the domain knowledge 21. The task decomposition unit 30 generates a language label indicating the content of each piece of subtask data, generates a subtask file in which the language labels are assigned to the subtask data, and generates a list of subtask data as shown in Fig. 3 (step S5). The subtask display unit 51 displays this list on the screen display unit 54.

[0059] In the processing of step S6, when the user inputs a linguistic instruction (corresponding to the second linguistic instruction described above) from the linguistic instruction unit 12 of the input device 10 to generate a subtask tree with a hierarchical structure, the task decomposition unit 30 stores a hierarchical structure in which related subtasks are grouped, as shown in FIG. 4, in the subtask file 41. The subtask display unit 51 displays the hierarchical structure of subtasks in a tree format, as shown in FIG. 5 (step S6). The storage unit 40 stores the subtask file 41 generated by the task decomposition unit 30 (step S7). The user can view both the video of the robot time-series data and the subtask list displayed by the subtask display unit 51 on the screen display unit 54.

[0060] Next, the subtask recommendation unit 60 estimates the confidence level of each subtask data (step S8), and the subtask display unit 51 displays the subtask data with a priority according to the confidence level (step S9). Specifically, under the control of the subtask display unit 51, the screen display unit 54 displays the subtask data with a lower confidence level first. Thereafter, the annotation device 1 starts the process again from step S2.

[0061] After the list of subtask data is generated in steps S2 to S9, the process returns to step S2. If the user has not given a language input instruction for the task (No in step S2), the task decomposition unit 30 determines whether the user has performed an operation to select specific subtask data (for example, clicking on subtask data with the mouse of the operation device 11) (step S10). If there has been no operation to select subtask data (No in step S10), the annotation device 1 determines whether annotation has ended (step S11). If annotation has ended (Yes in step S11), the annotation device 1 ends the process, and if annotation has not ended (No in step S11), the annotation device 1 starts the process again from step S2.

[0062] If a subtask data selection operation is performed (Yes in step S10), the subtask display unit 51 displays the start time and end time of the selected subtask data and the robot time-series data (step S12). Specifically, the GUI portion of Fig. 1 shows this display state. Next, the data input unit 52 launches an input form that accepts edited content for the subtask name, i.e., the language label (step S13).

[0063] Data input unit 52 determines whether a new language label has been input by the user via operation device 11 of input device 10 (step S14). If a new language label has been input (Yes in step S14), data input unit 52 edits the language label and updates subtask file 41, and storage unit 40 stores the updated subtask file 41 (step S15).

[0064] On the other hand, if no new language label has been input (No in step S14), data input unit 52 further determines whether the user has input new timestamps (start time and end time) for the subtask from operation device 11 of input device 10 (step S16). If a new timestamp has been input (Yes in step S16), data input unit 52 edits the time of the subtask, i.e., the timestamp, and updates subtask file 41, and storage unit 40 stores the updated subtask file 41 (step S17).

[0065] After step S15, or if no new timestamp is input for the subtask in step S16 (No in step S16), or after step S17, the process returns to step S8. In step S8, subtask recommendation unit 60 estimates the confidence level for the updated subtask file 41.

[0066] The annotation device 1 according to this embodiment has excellent interactivity. Interactivity means that the display and internal state of the system are updated in real time in response to the user's editing operations. The user can proceed with the work while checking at least the results of editing the language labels on the spot, enabling efficient and accurate annotation.

[0067] As a result, the present disclosure describes at least the following: Note that the components in parentheses correspond to those in the above-described embodiment, but are not limited to these.

[0068] (1) a task decomposition unit (30) that, in response to a linguistic instruction including a task specification input in a natural language, decomposes robot time-series data relating to the execution of the task into a plurality of subtask data, generates a linguistic label for each of the plurality of subtask data indicating the content of the subtask data, and generates a subtask file in which the linguistic label is assigned to the subtask data; A storage unit (40) that stores the subtask file. Annotation device (1).

[0069] As a result, the task decomposition unit decomposes the robot time-series data into subtask data, generates language labels, and assigns them to the subtask data, so that language labels can be automatically assigned to each subtask data in the robot time-series data, thereby reducing the time and cost required to build language-labeled robot data for machine learning.

[0070] (2) a display control unit (subtask display unit 51) that displays at least the language label of the subtask data in an editable manner; an input unit (data input unit 52) ​​that receives at least the edited content of the language label and updates the subtask file; the display control unit performs a display based on the updated subtask file. An annotation device according to (1).

[0071] This allows users to edit the assigned language labels, which significantly reduces the time and cost required compared to manually assigning language labels from scratch.

[0072] (3) the task decomposition unit stores a hierarchical structure in which related subtask data is grouped in the subtask file; the display control unit displays the hierarchical structure of the subtask data in a tree format. An annotation device according to (2).

[0073] This allows the hierarchical structure of the subtask data to be displayed in a tree format, making it easier to intuitively grasp related subtask data and to easily check and edit language labels.

[0074] (4) A data control unit (53) that controls the playback of the robot time-series data in response to a user operation. An annotation device according to (2).

[0075] This allows the user to edit the language labels while playing back the robot time-series data, thereby enabling efficient confirmation and editing of the language labels.

[0076] (5) The robot time-series data is multimodal sensor data including at least one of video, audio, force, and tactile data. An annotation device according to (1).

[0077] This allows it to handle tasks involving at least one of video, audio, force, and touch.

[0078] (6) The task decomposition unit generates the subtask data candidates by using a task decomposition model that combines natural language processing technology and a machine learning model and by referring to a knowledge base according to the type of robot or the domain of the task. An annotation device according to (1).

[0079] This makes it possible to generate appropriate subtask data candidates.

[0080] (7) The display control unit displays the tree by expanding or collapsing it in response to a user operation. An annotation device according to (3).

[0081] This allows the tree form of the hierarchical structure to be changed and displayed according to the user's wishes.

[0082] (8) The input unit accepts settings of start times and end times of the subtask data by a user, and updates the subtask file in accordance with the settings. the display control unit performs a display based on the updated subtask file in conjunction with the reproduction of the robot time-series data by the data control unit. An annotation device according to (4).

[0083] This allows the updated subtask file to be displayed when the robot time-series data is played back.

[0084] (9) A subtask recommendation unit (60) estimates a degree of certainty indicating the feasibility of each subtask data from the robot time-series data and the task information, and changes the presentation mode of the subtask data depending on the degree of the degree of certainty. An annotation device according to any one of (1) to (8).

[0085] This allows specific subtask data to be presented with priority depending on the feasibility.

[0086] (10) In response to a linguistic instruction including a task specification input in natural language, decomposing robot time-series data regarding the execution of the task into a plurality of subtask data; generating a linguistic label indicating the content of each of the plurality of subtask data; generating a subtask file in which the language label is assigned to the subtask data; storing the subtask file; Annotation method.

[0087] As a result, the task decomposition unit decomposes the robot time-series data into subtask data, generates language labels, and assigns them to the subtask data, so that language labels can be automatically assigned to each subtask data in the robot time-series data, thereby reducing the time and cost required to build language-labeled robot data for machine learning.

[0088] (11) a step of decomposing robot time-series data relating to the execution of a task into a plurality of subtask data in response to a linguistic instruction including a task specification input in a natural language; generating a linguistic label indicating the content of each of the plurality of subtask data; generating a subtask file in which the language label is assigned to the subtask data; storing the subtask file; An annotation program that causes a computer to execute the above.

[0089] As a result, the task decomposition unit decomposes the robot time-series data into subtask data, generates language labels, and assigns them to the subtask data, so that language labels can be automatically assigned to each subtask data in the robot time-series data, thereby reducing the time and cost required to build language-labeled robot data for machine learning. [Industrial Applicability]

[0090] The present disclosure is useful as an annotation device, an annotation method, and an annotation program that can reduce the time and cost required to build language-labeled robot data for machine learning. [Explanation of symbols]

[0091] 1. Annotation device 10 Input Devices 11 Operating devices 12 Language instruction section 20 DB 21 Domain Knowledge 22 Robot Data 30 Task decomposition section 40 Storage section 41 Subtask Files 50 GUI 51 Subtask display section (display control section) 52 Data input section (input section) 53 Data control section 54 Screen display section 60 Subtask Recommendation

Claims

1. a task decomposition unit that, in response to a linguistic instruction including a task specification input in a natural language, decomposes robot time-series data related to the execution of the task into a plurality of subtask data, generates a linguistic label indicating the content of each of the plurality of subtask data, and generates a subtask file in which the linguistic label is assigned to the subtask data; A storage unit that stores the subtask file. Annotation device.

2. a display control unit that displays at least the language label of the subtask data in an editable manner; an input unit that receives at least the edited content of the language label and updates the subtask file; the display control unit performs a display based on the updated subtask file. The annotation device according to claim 1 .

3. the task decomposition unit stores a hierarchical structure in the subtask file, in which related subtask data are grouped; the display control unit displays the hierarchical structure of the subtask data in a tree format. The annotation device according to claim 2 .

4. a data control unit that controls playback of the robot time-series data in response to a user operation; The annotation device according to claim 2 .

5. The robot time-series data is multimodal sensor data including at least one of video, audio, force, and tactile data. The annotation device according to claim 1 .

6. the task decomposition unit generates the subtask data candidates by using a task decomposition model that combines natural language processing technology and a machine learning model and by referring to a knowledge base according to the type of robot or the domain of the task; The annotation device according to claim 1 .

7. The display control unit displays the tree by expanding or collapsing it in response to a user operation. The annotation device according to claim 3 .

8. the input unit accepts settings of start times and end times of the subtask data from a user, and updates the subtask file in accordance with the settings; the display control unit performs a display based on the updated subtask file in conjunction with the reproduction of the robot time-series data by the data control unit. The annotation device according to claim 4 .

9. a subtask recommendation unit that estimates a degree of certainty indicating the feasibility of each subtask data from the robot time-series data and the task information, and changes a presentation mode of the subtask data in accordance with the degree of the degree of certainty; The annotation device according to any one of claims 1 to 8.

10. In response to a linguistic instruction including a task specification input in natural language, decompose robot time-series data regarding the execution of the task into a plurality of subtask data; generating a linguistic label indicating the content of each of the plurality of subtask data; generating a subtask file in which the language label is assigned to the subtask data; storing the subtask file; Annotation method.

11. a step of decomposing robot time-series data relating to the execution of a task into a plurality of subtask data in response to a linguistic instruction including a task specification input in a natural language; generating a linguistic label indicating the content of each of the plurality of subtask data; generating a subtask file in which the language label is assigned to the subtask data; storing the subtask file; An annotation program that causes a computer to execute the above.

Images