Data augmentation method, device, equipment and storage medium

Abstract

The application discloses a data enhancement method, device, equipment and storage medium, and belongs to the technical field of computers. The method specifically comprises the following steps: acquiring teaching data, including visual data, robot state data, action sequences and language instructions; constructing a task hierarchical semantic graph based on the language instructions, wherein the graph comprises a root task node, a subtask node and an atomic action node; obtaining the initial state, action sequence, termination state and completion criterion of the root task node, subtask node and atomic action node based on the visual data, robot state data and action sequences; obtaining each task-level language instruction enhancement data based on the root task node, subtask node and atomic action node; and determining training sample data based on the initial state, action sequence, termination state and completion criterion of the root task node, subtask node and atomic action node and each task-level language instruction enhancement data, so as to train a model.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to the fields of big data processing technology, large model training technology, and more particularly to a data augmentation method, apparatus, device, and storage medium. Background Technology

[0002] Embodied intelligence refers to artificial intelligence technology in which intelligent agents perceive the physical environment through sensors such as vision, language, touch, posture, and force, and act on the physical environment through robotic arms, mobile chassis, dexterous hands, or other actuators to complete specified tasks. The Vision-Language-Action (VLA) model is an important model type in the field of embodied intelligence.

[0003] Currently, when training VLA models in related technologies, the VLA model typically receives training sample data such as scene images or video sequences, natural language task instructions, robot's own state, and historical observations and actions. After visual encoding, language encoding, and multimodal fusion processing, it outputs robot actions. However, the samples used for model training have many defects and limitations, resulting in poor language understanding, state-action correspondence, and task execution stability of the model for complex embodied tasks. Summary of the Invention

[0004] This application provides a data augmentation method, apparatus, device, and storage medium, which can solve the problem of poor learning performance of VLA models due to defects in training data. The technical solution is as follows: In a first aspect, a data augmentation method is provided, the method comprising: Acquire teaching data to be processed, the teaching data including visual data, robot state data, action sequences, and language instructions corresponding to the action sequences; Based on the language instructions, a task-level semantic graph is constructed, which includes a root task node, sub-task nodes, and atomic action nodes. Based on the visual data, robot state data, and action sequence, and using a preset alignment strategy, the task start state data, segment action sequence, task termination state data, and task completion criteria of the root task node, sub-task node, and atomic action node are obtained respectively. Based on the root task node, sub-task node, and atomic action node, and using a preset language instruction enhancement strategy, root task-level language instruction enhancement data, sub-task-level language instruction enhancement data, and action-level language instruction enhancement data are obtained respectively. Based on the task start state data, fragment action sequence, task termination state data and task completion criteria of the root task node, sub-task node, and atomic action node, as well as the root task-level language instruction enhancement data, sub-task-level language instruction enhancement data and action-level language instruction enhancement data, the training sample data is determined. Based on the training sample data, the visual language action model to be trained is trained.

[0005] In one possible implementation, constructing a task-level semantic graph based on the language instructions includes: The language instructions are subjected to task semantic parsing to obtain task element information; Based on the task element information, a root task node, sub-task nodes, atomic action nodes, and the relationships between nodes are generated. Based on the root task node, sub-task nodes, atomic action nodes, and the relationships between nodes, a task-level semantic graph is constructed.

[0006] In one possible implementation, the step of obtaining task start state data, segment action sequences, task termination state data, and task completion criteria for the root task node, sub-task nodes, and atomic action nodes based on the visual data, robot state data, and action sequences, using a preset alignment strategy, includes: Based on the task execution time intervals corresponding to the root task node, sub-task node, and atomic action node, the task execution segments corresponding to the root task node, sub-task node, and atomic action node are determined respectively. Based on the visual data, robot state data, and action sequence, the task start state data, segment action sequence, and task end state data of each task execution segment are extracted respectively. Determine the task types of the root task node, sub-task nodes, and atomic action nodes respectively; Based on the preset correspondence between task types and criteria, the task completion criteria of the root task node, sub-task node, and atomic action node are obtained respectively. In one possible implementation, the step of obtaining root-task-level language instruction enhancement data, sub-task-level language instruction enhancement data, and action-level language instruction enhancement data based on the root task node, sub-task node, and atomic action node using a preset language instruction enhancement strategy includes: Based on the root task node, sub-task node, and atomic action node, obtain root task-level language instructions, sub-task-level language instructions, and action-level language instructions. Semantic transformation processing is performed on the root task-level language instructions, subtask-level language instructions, and action-level language instructions respectively to obtain candidate root task-level language instruction variants, candidate subtask-level language instruction variants, and candidate action-level language instruction variants. Based on the task-level language instructions, subtask-level language instructions, and action-level language instructions, the root candidate task-level language instruction variants, candidate subtask-level language instruction variants, and candidate action-level language instruction variants are respectively verified to obtain the root task-level language instruction variants, subtask-level language instruction variants, and action-level language instruction variants. Based on root task-level language instructions, subtask-level language instructions, action-level language instructions, root task-level language instruction variants, subtask-level language instruction variants, and action-level language instruction variants, respectively, root task-level language instruction enhancement data, subtask-level language instruction enhancement data, and action-level language instruction enhancement data are obtained.

[0007] In one possible implementation, training sample data is determined based on the task start state data, fragment action sequence, task termination state data, and task completion criteria of the root task node, sub-task nodes, and atomic action nodes, as well as root task-level language instruction enhancement data, sub-task-level language instruction enhancement data, and action-level language instruction enhancement data. This includes: Based on the enhanced language instruction data at the root task level and the starting state data, fragment action sequence, and task termination state data of the root task node, root task level sample data is obtained. Subtask-level sample data is obtained based on subtask-level language instruction enhancement data and subtask node start state data, fragment action sequence, and task termination state data. Based on the action-level language instruction enhancement data and the start state data, fragment action sequence and task termination state data of atomic action nodes, action-level sample data are obtained. Training sample data is obtained based on the root task-level sample data, sub-task-level sample data, and action-level sample data.

[0008] In one possible implementation, the training process of the visual language action model to be trained based on the training sample data includes: Based on the training sample data, the root task-level sample data, sub-task-level sample data, and action-level sample data are selected and processed according to a preset selection ratio. Based on the results of the selection process, the visual language action model to be trained is trained to obtain training evaluation information. In response to the training evaluation information indicating a high-risk situation, the preset selection ratio is adjusted. Based on the training sample data, and according to the results of the adjustment process, the root task-level sample data, sub-task-level sample data, and action-level sample data are updated and selected, so as to train the visual language action model to be trained based on the results of the update and selection process.

[0009] Secondly, a data augmentation apparatus is provided, the apparatus comprising: The acquisition unit is used to acquire teaching data to be processed, the teaching data including visual data, robot state data, action sequence, and language instructions corresponding to the action sequence; The construction unit is used to construct a task-level semantic graph based on the language instructions. The task-level semantic graph includes a root task node, sub-task nodes, and atomic action nodes. The alignment unit is used to obtain the task start state data, fragment action sequence, task termination state data and task completion criteria of the root task node, sub-task node and atomic action node respectively based on the visual data, robot state data and action sequence and using a preset alignment strategy. The enhancement unit is used to obtain root task-level language instruction enhancement data, sub-task-level language instruction enhancement data, and action-level language instruction enhancement data respectively based on the root task node, sub-task node, and atomic action node, using a preset language instruction enhancement strategy. The determining unit is used to determine training sample data based on the task start state data, fragment action sequence, task termination state data and task completion criteria of the root task node, sub-task node, and atomic action node, as well as the root task-level language instruction enhancement data, sub-task-level language instruction enhancement data and action-level language instruction enhancement data. The training unit is used to train the visual language action model to be trained based on the training sample data.

[0010] Thirdly, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the aspects and any possible implementations described above.

[0011] Fourthly, an electronic device is provided, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods described above and any possible implementations.

[0012] Fifthly, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the aspects and any possible implementations described above.

[0013] The beneficial effects of the technical solution provided in this application include at least the following: As can be seen from the above technical solution, the embodiments of this application can acquire teaching data to be processed, which includes visual data, robot state data, action sequences, and language instructions corresponding to the action sequences. Based on the language instructions, a task-level semantic graph is constructed, which includes root task nodes, sub-task nodes, and atomic action nodes. Based on the visual data, robot state data, and action sequences, and using a preset alignment strategy, task start state data, fragment action sequences, task termination state data, and task completion criteria of the root task nodes, sub-task nodes, and atomic action nodes are obtained respectively. Based on the root task nodes, sub-task nodes, and atomic action nodes, and using a preset language instruction enhancement strategy, root task-level language instruction enhancement data, sub-task-level language instruction enhancement data, and action-level language instruction enhancement data are obtained respectively. Based on the task start state data, fragment action sequences, and task termination state data of the root task nodes, sub-task nodes, and atomic action nodes, and the task completion criteria of the sub-task nodes, sub-task nodes, and atomic action nodes are obtained respectively. The system uses sequence data, task termination state data, and task completion criteria, along with root task-level language instruction enhancement data, sub-task-level language instruction enhancement data, and action-level language instruction enhancement data, to determine training sample data. Based on this training sample data, the visual language action model to be trained is trained. Since it is possible to generate multiple language instruction enhancement data with different levels of granularity from the original language instructions according to the hierarchical structure of the robot's embodied task without significantly increasing the cost of collecting real robot teaching data, and to combine the language instruction enhancement data with the robot's initial state, action sequence, termination state, and task completion criteria during execution to obtain training sample data, a five-element training sample data structure of multi-granularity instruction—initial state—action sequence—termination state—completion criterion is automatically constructed for training the visual language action model. This improves the visual language action model's language understanding ability, state-action correspondence ability, and task execution stability for complex embodied tasks.

[0014] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating a data augmentation method provided in one embodiment of this application; Figure 2 This is a flowchart illustrating a data augmentation method provided in another embodiment of this application; Figure 3 This is a schematic diagram of the architecture of a data augmentation method provided in an embodiment of this application; Figure 4 This is a structural block diagram of a data enhancement apparatus provided in another embodiment of this application; Figure 5 This is a block diagram of an electronic device used to implement the data enhancement method of the embodiments of this application. Detailed Implementation

[0017] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0018] Obviously, the described embodiments are only some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0019] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0020] Currently, most related technologies rely on VLA training methods based on human-taught data. These methods involve: first, collecting video, images, robot states, and action sequences during task execution; second, assigning a natural language task instruction to each segment of teaching data; third, inputting the images, language instructions, and action sequences into the VLA model; fourth, training the VLA model through behavior cloning or supervised learning to learn the mapping relationship from visual and language inputs to action outputs; and finally, during actual execution, inputting new images and language instructions into the trained VLA model to output robot actions. This method typically associates a complete language instruction with an entire action sequence, lacking a task-stage and sub-task-level correspondence between the language instruction and the action sequence.

[0021] However, in this method, language instructions are usually expressed in a flat manner, lacking a hierarchical task structure. A complete task corresponds to only one overall instruction, and the model cannot distinguish between the overall goal, stage goals, and atomic actions. The model struggles to learn the order of subtasks and stage transitions in complex tasks. As the number of task steps increases, the action prediction error gradually accumulates, and the success rate and stability of multi-step tasks decrease. Moreover, language enhancement in related technologies mainly involves synonym replacement, resulting in insufficient semantic coverage.

[0022] Therefore, there is an urgent need for a data augmentation method that can effectively achieve task-level representation and multi-dimensional enhancement of language instructions, thereby ensuring the stability and reliability of the inference performance of the trained VLA model.

[0023] Please refer to Figure 1 This illustration shows a flowchart of a data augmentation method provided in one embodiment of this application. Specifically, the data augmentation method may include: Step 101: Obtain the teaching data to be processed. The teaching data includes visual data, robot state data, action sequences, and language instructions corresponding to the action sequences.

[0024] Step 102: Based on the language instructions, construct a task-level semantic graph, which includes root task nodes, sub-task nodes, and atomic action nodes.

[0025] Step 103: Based on the visual data, robot state data, and action sequence, and using a preset alignment strategy, obtain the task start state data, segment action sequence, task termination state data, and task completion criteria for the root task node, sub-task node, and atomic action node, respectively.

[0026] Step 104: Based on the root task node, sub-task node, and atomic action node, obtain root task-level language instruction enhancement data, sub-task-level language instruction enhancement data, and action-level language instruction enhancement data respectively using a preset language instruction enhancement strategy.

[0027] Step 105: Based on the task start state data, fragment action sequence, task termination state data, and task completion criteria of the root task node, sub-task node, and atomic action node, as well as the root task-level language instruction enhancement data, sub-task-level language instruction enhancement data, and action-level language instruction enhancement data, determine the training sample data.

[0028] Step 106: Based on the training sample data, train the visual language action model to be trained.

[0029] It should be noted that teaching data can be data generated when the robot performs one or more embodied tasks. Teaching data may also include action completion labels, task success labels, and task time markers.

[0030] It should be noted that visual data can include multiple frames of scene images or video sequences. Robot state data can include joint angles, joint velocities, end effector pose, gripper opening / closing state, mobile chassis pose, target object position, and environmental obstacle information. Action sequences can be sequences of actions performed by the robot to perform embodied tasks. Action sequences can include robotic arm joint positions, robotic arm joint velocities, robotic arm end effector pose, gripper opening / closing commands, mobile robot linear velocity, mobile robot angular velocity, action duration, and multi-step action sequences. Action sequences can be historical action sequences. Language commands can be natural language task commands.

[0031] It is important to understand that, here, "robot" can include wheeled, dual-arm mobile robots.

[0032] In this way, without significantly increasing the cost of collecting real robot teaching data, multiple language instruction enhancement data with different levels of granularity can be generated from the original language instructions according to the hierarchical structure of the robot's embodied task. The language instruction enhancement data is then combined with the robot's initial state, action sequence, termination state, and task completion criteria during execution to obtain training sample data. This automatically constructs a five-element training sample data consisting of multi-granularity instructions, initial state, action sequence, termination state, and completion criteria, which can be used to train the visual language action model. This improves the visual language action model's language understanding ability, state-action correspondence ability, and task execution stability for complex embodied tasks.

[0033] Optionally, in one possible implementation of this embodiment, in step 102, firstly, the language instructions can be subjected to task semantic parsing processing to obtain task element information. Secondly, based on the task element information, root task nodes, sub-task nodes, atomic action nodes, and the relationships between nodes can be generated. Thirdly, based on the root task nodes, sub-task nodes, atomic action nodes, and the relationships between nodes, a task hierarchical semantic graph can be constructed.

[0034] In this implementation, the task element information may include at least one of the following: task objective, task object, object attributes, spatial relationship, action type, preconditions, postconditions, safety constraints, execution order, and task completion conditions.

[0035] For example, if the language instruction is "Put the red square into the blue box", the parsed task element information may include: Task objective: Complete the placement; Target of manipulation: Red square; Target area: Blue box; Action types: grab, move, release; Spatial constraints: The red cube is located inside the blue box; Prerequisite: The robotic arm can reach the red square; Postcondition: The red block is stably inside the blue box.

[0036] In one specific implementation of this approach, based on the task element information, a root task node, one or more sub-task nodes, one or more atomic action nodes, attributes of each node, and relationships between nodes are generated. Furthermore, a task-level semantic graph can be constructed based on the root task node, one or more sub-task nodes, one or more atomic action nodes, attributes of each node, and relationships between nodes.

[0037] In this implementation, the task-level semantic graph may include a set of task nodes, a set of node relationships, and a set of node attributes.

[0038] For example, a task-level semantic graph can be represented as: ,in: Represents a set of task nodes; Represents a set of node relationships; A task-level semantic graph represents a set of node attributes. It includes at least: a root task node; one or more subtask nodes; one or more atomic action nodes; node attributes; and relationships between nodes.

[0039] In this implementation, the root task node represents the complete task, including complete task information. Subtask nodes represent intermediate tasks with phased goals, and may include intermediate task information. Atomic action nodes represent short-lived, executable specific actions.

[0040] In this implementation, the relationships between nodes in the task-level semantic graph can include at least one of the following: hierarchical inclusion relationship, execution order relationship, conditional dependency relationship, and state transition relationship.

[0041] In this way, the task-level semantic graph can be constructed by processing the results of task semantic parsing of language instructions. The tasks in the language instructions are decomposed into a three-level structure of root task, sub-task, and atomic action. By hierarchical semantic instructions, the training data is enriched so that the model can learn the feature information such as task information, state data and corresponding action training at different levels during subsequent model training.

[0042] Optionally, in one possible implementation of this embodiment, in step 103, firstly, based on the task execution time intervals corresponding to the root task node, sub-task node, and atomic action node, the task execution segments corresponding to the root task node, sub-task node, and atomic action node can be determined respectively. Secondly, based on the visual data, robot state data, and action sequence, the task start state data, segment action sequence, and task end state data of each task execution segment can be extracted respectively. Thirdly, the task type of the root task node, sub-task node, and atomic action node can be determined respectively. Fourthly, based on the preset task type and criterion correspondence relationship, the task completion criteria of the root task node, sub-task node, and atomic action node can be obtained respectively.

[0043] In a specific implementation of this approach, based on the task time markers in the teaching data, the task execution time interval corresponding to the root task node, the task execution time interval corresponding to each sub-task node, and the task execution time interval corresponding to each atomic action node can be determined respectively.

[0044] Here, the root task node, sub-task nodes, and atomic action nodes can be collectively referred to as task nodes.

[0045] One specific implementation process involves determining the start and end times of each task node based on the task time stamps in the teaching data. Then, based on the start and end times of each task node, the task execution time interval for each task node is determined, and the task execution time interval for each task node is defined as the task execution segment for each task node.

[0046] For example, for task nodes Determine its execution time range: ,in, This indicates the time when the task node starts executing. This indicates the time when the task node finishes execution. Here, the... The execution time interval can be used as The task execution segment.

[0047] It is understandable that task execution segments can also be determined based on other methods. For example, the task execution segment for each task node can be determined based on motion change detection results, end effector pose changes, target object visual state changes, task completion criteria, or matching results between graph nodes and action sequences. No specific limitations are imposed here; the appropriate determination method can be selected based on actual business needs.

[0048] In this implementation, there can be multiple subtask nodes and multiple atomic action nodes.

[0049] In another specific implementation of this approach, for each task node, based on the visual data, robot state data, and action sequence, the task start state data, action sequence, and task termination state data of the task execution segment of each task node are extracted respectively. Thus, the task start state data, action sequence, and task termination state data of the root task node, the task start state data, action sequence, and task termination state data of each sub-task node, and the task start state data, action sequence, and task termination state data of each atomic action node can be obtained.

[0050] In this implementation, the action sequence can be the sequence of actions performed by the robot within a task execution segment.

[0051] In this implementation, the task start state data includes at least one of the following: start image, start depth information, start joint state, start end pose, start gripper state, and the relative position of the target object and the robotic arm.

[0052] In this implementation, the task termination status data includes at least one of the following: termination image, termination depth information, termination joint state, termination end pose, termination gripper state, and relative position of the target object and the robotic arm.

[0053] One specific implementation process involves extracting the task start state data, segment action sequence, and task termination state data of the root task node based on the visual data, robot state data, action sequence, and task execution segment corresponding to the root task node.

[0054] Another specific implementation process involves extracting the task start state data, segment action sequence, and task termination state data for each sub-task node based on the visual data, robot state data, action sequence, and task execution segment corresponding to each sub-task node.

[0055] Another specific implementation process involves extracting the task start state data, segment action sequence, and task termination state data for each atomic action node based on the visual data, robot state data, action sequence, and task execution segment corresponding to each atomic action node.

[0056] For example, for each task node, the robot state data and action sequence within its execution segment are extracted. The starting state Si can be the state at the beginning of the execution segment or at several consecutive moments near the beginning, and the ending state Si' can be the state at the end of the execution segment or at several consecutive moments near the end. Action sequence Time interval The set of actions within: ,in, and It can be the starting action or the ending action in an action sequence.

[0057] In this implementation, task types can include grasping task types, placement task types, and movement task types. The preset task type-criteria correspondence can be a correspondence between each task type and the completion criterion for that task. This preset task type-criteria correspondence can include a grasping task type-criteria correspondence and a placement task type-criteria correspondence. The criterion can be the basis for determining whether the robot has completed the task, i.e., the completion criterion.

[0058] Here, the completion criteria for the grasping task type can include the gripper being closed, the target object and the gripper remaining relatively stable, the target object moving with the end effector, and the distance between the target object and the table exceeding a preset value. When all these completion criteria are met, the robot can be considered to have completed the grasping task.

[0059] The completion criteria for the placement task type may include the target object being located inside the target container, the distance between the target object and the center of the target area being less than a preset threshold, the target object's speed being lower than a preset threshold, the gripper being open, and the placement status being met for multiple consecutive frames.

[0060] The completion criteria for the mobile task type may include a position deviation less than a preset first threshold and a heading angle deviation less than a preset second threshold.

[0061] In this way, by determining the task start state data, fragment action sequence, task termination state data, and task completion criteria corresponding to different levels of tasks such as root task nodes, sub-task nodes, and atomic action nodes, the model can learn the tasks and actions with different hierarchical relationships during subsequent training, thereby further improving the performance of the training model.

[0062] It should be noted that the specific implementation process provided in this implementation method can be combined with various specific implementation processes provided in the foregoing implementation methods to realize the data augmentation method of this embodiment. Detailed descriptions can be found in the relevant content of the foregoing implementation methods, and will not be repeated here.

[0063] Optionally, in one possible implementation of this embodiment, in step 104, firstly, root task-level language instructions, sub-task-level language instructions, and action-level language instructions can be obtained based on the root task node, sub-task node, and atomic action node. Secondly, semantic transformation processing can be performed on the root task-level language instructions, sub-task-level language instructions, and action-level language instructions respectively to obtain candidate root task-level language instruction variants, candidate sub-task-level language instruction variants, and candidate action-level language instruction variants. Thirdly, based on the task-level language instructions, sub-task-level language instructions, and action-level language instructions, verification processing can be performed on the root candidate task-level language instruction variants, candidate sub-task-level language instruction variants, and candidate action-level language instruction variants respectively to obtain root task-level language instruction variants, sub-task-level language instruction variants, and action-level language instruction variants. Fourthly, root task-level language instruction enhancement data, sub-task-level language instruction enhancement data, and action-level language instruction enhancement data can be obtained based on the root task-level language instructions, sub-task-level language instructions, action-level language instructions, root task-level language instruction variants, sub-task-level language instruction variants, and action-level language instruction variants respectively.

[0064] In a specific implementation of this approach, based on task element information, the language instructions of the root task node, the language instructions of the sub-task nodes, and the language instructions of the atomic action nodes can be determined. The language instructions of the root task node are taken as root task-level language instructions, the language instructions of the sub-task nodes are taken as sub-task-level language instructions, and the language instructions of the atomic action nodes are taken as action-level language instructions.

[0065] For example, the root task is: put the red block into the blue box; subtask 1: locate the red block; subtask 2: grab the red block; subtask 3: move the red block; subtask 4: release the red block; subtask 5: confirm placement is complete.

[0066] In another specific implementation of this approach, preset transformation rules can be used to perform semantic transformation processing on the root task-level language instructions, subtask-level language instructions, and action-level language instructions respectively, in order to obtain candidate root task-level language instruction variants, candidate subtask-level language instruction variants, and candidate action-level language instruction variants.

[0067] In this implementation, the preset transformation rules can be used to transform language instructions based on multiple transformation dimensions while keeping the task objective, task object and key execution constraints consistent.

[0068] Here, the transformation dimension can be multi-dimensional semantic enhancement. Transformation dimensions include perspective transformation, abstraction level transformation, spatial relationship expression transformation, constraint condition transformation, temporal expression transformation, and sentence reference transformation.

[0069] Here, perspective transformation refers to the shift between the perspective of action execution, the perspective of the target object, or the perspective of the result state. Abstraction level transformation involves the shift between high-level abstract descriptions, mid-level task descriptions, and low-level action descriptions. Spatial relationship expression transformation involves the transformation of expressions of spatial location, direction, distance, inclusion relationships, and contact relationships; constraint condition transformation involves the addition or transformation of environmental conditions, obstacle avoidance conditions, and operational constraints. Temporal sequence expression transformation involves the transformation of expressions of the sequential relationship of actions and the conditions for the completion of stages. Sentence referential transformation involves the transformation of active sentences, passive sentences, referential forms, and word order.

[0070] In another specific implementation of this approach, based on the task-level language instructions, subtask-level language instructions, and action-level language instructions, and using preset verification rules, the root candidate task-level language instruction variants, candidate subtask-level language instruction variants, and candidate action-level language instruction variants are respectively verified to obtain the root task-level language instruction variants, subtask-level language instruction variants, and action-level language instruction variants.

[0071] In one implementation scenario, language instructions can include information such as task objectives, task objects, target regions, key spatial relationships, preconditions, postconditions, and action executability parameters. For any task node, the task objectives, task objects, target regions, key spatial relationships, preconditions, postconditions, and action executability parameters of the language instructions and candidate language instruction variants can be checked for consistency to obtain language instruction variants that pass the check.

[0072] Understandably, here, based on the attributes and relationships of nodes in the task-level semantic graph, the task objective, task object, and key execution constraints of the node are used as generation constraints. Different instruction variants with different perspectives, different levels of abstraction, different spatial relationship descriptions, different constraints, different temporal expressions, and different sentence structures are generated for the same node. At the same time, the original instruction is used as a reference constraint. When the instruction variant changes the task objective or key execution constraints, the instruction variant can be marked as invalid and not used for training, which can avoid the generated results from deviating from the semantics.

[0073] In this implementation, the preset verification rules include at least one of the following: graph constraint rule verification, large language model structure verification, reverse generation verification, and cross-verification of variants of the same node.

[0074] Here, graph constraint rule verification can involve parsing language instruction variants into task elements and comparing them item by item with the task objective, task object, key spatial relationships, preconditions, and postconditions of the corresponding nodes in the graph; any inconsistency renders the graph invalid. Large language model structured verification involves inputting the original language instruction and its variants into the large language model and outputting the consistency judgment result and the reason for any inconsistency. Reverse generation verification involves generating a task graph or execution plan from the language instruction variants and comparing its structural consistency with the original graph. Cross-verification of variants at the same node involves comparing the semantic consistency of multiple language instruction variants generated from the same task node pairwise, retaining only mutually consistent variants.

[0075] In this way, language instruction variants can be generated based on transformations of different dimensions. The same task action can be corresponding to multiple natural language instructions with different expressions. This allows the model to fully learn the relationship between natural language and actions, effectively improving the model's ability to adapt to user-differentiated language instructions, reducing the standardization requirements of human-computer interaction instructions, greatly improving the generalization ability of natural language instructions, and adapting to diverse human-computer interaction expressions.

[0076] It should be noted that the specific implementation process provided in this implementation method can be combined with various specific implementation processes provided in the foregoing implementation methods to realize the data augmentation method of this embodiment. Detailed descriptions can be found in the relevant content of the foregoing implementation methods, and will not be repeated here.

[0077] Optionally, in one possible implementation of this embodiment, in step 105, firstly, root task-level sample data can be obtained based on root task-level language instruction enhancement data and the start state data, fragment action sequence, and task termination state data of the root task node. Secondly, sub-task-level sample data can be obtained based on sub-task-level language instruction enhancement data and the start state data, fragment action sequence, and task termination state data of the sub-task node. Thirdly, action-level sample data can be obtained based on action-level language instruction enhancement data and the start state data, fragment action sequence, and task termination state data of the atomic action node. Finally, training sample data can be obtained based on the root task-level sample data, sub-task-level sample data, and action-level sample data.

[0078] In a specific implementation of this method, in step 106, firstly, based on the training sample data, the root task-level sample data, sub-task-level sample data, and action-level sample data can be selected according to a preset selection ratio. Secondly, based on the result of the selection process, the visual language action model to be trained can be trained to obtain training evaluation information. Thirdly, in response to the training evaluation information indicating a high-risk situation, the preset selection ratio can be adjusted. Fourthly, based on the training sample data and according to the result of the adjustment process, the root task-level sample data, sub-task-level sample data, and action-level sample data can be updated, and based on the result of the updated selection process, the visual language action model to be trained can be trained.

[0079] Here, root task-level sample data, subtask-level sample data, and action-level sample data can represent sample data at different granularities.

[0080] In this implementation, the preset selection ratio may include a selection ratio based on equal sampling and a selection ratio based on weighted sampling.

[0081] In one specific implementation, in response to the training evaluation information indicating a high-risk situation, the proportion of action-level sample data can be increased to obtain an adjusted preset selection ratio. Based on the training sample data, the root task-level sample data, sub-task-level sample data, and action-level sample data are updated and selected according to the adjusted preset selection ratio.

[0082] In another specific implementation of this method, in response to the training evaluation information being of medium risk, the proportion of subtask-level sample data can be increased and the proportion of root task-level sample data can be adjusted to 0 to obtain the adjusted preset selection ratio.

[0083] In another specific implementation of this approach, in response to the low-risk situation of the training evaluation information, the proportion of root task-level sample data can be increased to obtain the adjusted preset selection ratio.

[0084] In another specific implementation of this approach, based on the results of the selection process, the visual language action model to be trained is trained to obtain training evaluation metrics. Based on these metrics, a preset weighted algorithm is used to obtain evaluation values. Based on these evaluation values, training evaluation information is obtained. The training evaluation information may include high-risk, low-risk, and medium-risk scenarios.

[0085] Here, training evaluation metrics may include at least one of the following: number of failures, chassis alignment deviation, dual-arm posture deviation, visual recognition confidence, and execution time deviation.

[0086] In this way, the selection of training sample data at different levels can be adjusted based on the evaluation information of model training, which can further improve the effect and effectiveness of model training.

[0087] It should be noted that the specific implementation process provided in this implementation method can be combined with various specific implementation processes provided in the foregoing implementation methods to realize the data augmentation method of this embodiment. Detailed descriptions can be found in the relevant content of the foregoing implementation methods, and will not be repeated here.

[0088] To better understand the methods of the embodiments of this application, the methods of the embodiments of this application will be described below in conjunction with the accompanying drawings and specific application scenarios.

[0089] Figure 2 This is a flowchart illustrating a data augmentation method provided in another embodiment of this application, as shown below. Figure 2 As shown. This data augmentation method may specifically include: Step 201: Obtain the robot's teaching data, which includes visual data, robot state data, action sequences, and language commands.

[0090] In this implementation, the robot can be a dual-arm wheeled mobile robot. This robot has the ability to move and avoid obstacles, move in all directions, and work collaboratively with both arms. It can adapt to work scenarios involving large, easily tipped-over, and irregularly shaped objects on a desktop that cannot be stably grasped by a single arm. It fully leverages the advantages of the wheeled chassis for all-around mobility and the advantages of the dual arms for collaborative gripping, posture correction, and collaborative handling to complete the task of organizing and storing various types of desktop objects across different locations.

[0091] For example, consider a dual-arm wheeled mobile robot performing tasks such as sorting and collaboratively transporting irregularly shaped objects on a desktop. This dual-arm wheeled mobile robot can be an industrial-grade interactive embodied wheeled robot with a total weight of 185kg. It has 26 active degrees of freedom, equipped with a standard 5-DOF (degrees of freedom) waist and leg structure and an omnidirectional wheeled mobile chassis, supporting omnidirectional translation, fixed-point parking, high-precision alignment, low-speed stable stopping, and all-area obstacle avoidance. The robot features two symmetrically arranged 7-DOF force-controlled arms, each with 7 degrees of freedom, supporting independent operation, bilateral collaborative gripping, relay transport, and compliant force control interaction, adaptable to precision operations on various irregularly shaped objects. The robot comes standard with a cross-shaped wrist force-controlled end effector, equipped with an adaptive flexible two-finger gripper, and distributed high-precision torque sensors throughout the arms, supporting joint impedance control, delicate external force perception, compliant contact feedback, and closed-loop gripping control. Each arm has a rated load of 18kg, and the peak load for both arms in collaboration can reach 50kg, enabling stable gripping of large, easily tipped, irregularly shaped objects. Equipped with a global RGB-D main camera on the robot's head, it is responsible for the overall layout of the work scene, coarse target localization, obstacle detection, and overall perception of the storage area. The left and right robotic arms each have embedded close-range RGB-D cameras in their wrists, which dynamically follow the arm to achieve sub-millimeter-level precise alignment of the grasping area, detection of the gripping contact surface, and posture deviation correction. It possesses a multi-modal collaborative perception system combining global head control and localized precision measurement by both wrists. It can be equipped with a standard NVIDIA Jetson Thor T5000 chip, with a local computing power of 2070 TFLOPS (FP4), enabling real-time onboard VLA embodied modeling, omnidirectional chassis trajectory planning, dual-arm collaborative force control calculation, and multi-sensor data fusion processing. It can complete the entire process of edge-side inference and control without external equipment.

[0092] The standardized desktop work scenario features three typical object types: large red cubes, slender blue cylinders, and lightweight, irregularly shaped green spheres. These objects are characterized by unstable centers of gravity, susceptibility to tipping over when grasped by a single arm, and irregular contact surfaces. This aligns with the robot's core advantages of dual-arm force control and collaborative gripping, as well as real-time posture correction. Furthermore, the dispersed placement of these targets complements the all-terrain mobility and positioning capabilities of the wheeled chassis. Three corresponding colored target storage boxes are distributed across different work areas on the desktop. Leveraging the robot's omnidirectional wheeled chassis mobility and precise dual-arm placement capabilities, these boxes autonomously collect and store objects across different areas.

[0093] Here, the teaching data for a single task can include: a global RGB image of the head at 30 frames per second, and local RGB images of the left and right wrists; real-time position, velocity, and torque data of 7 joints on each of the left and right arms; six-dimensional pose of the left and right end effectors and the relative pose difference between the two arms; opening and closing status, gripping torque, and contact force feedback data of the left and right grippers; independent control actions of the two arms and coordinated matching action sequences; data on the wheeled chassis movement speed, movement displacement, and heading angle; chassis alignment deviation, coordinated posture deviation of the two arms, and object posture offset data; success or failure labels for the overall task and each coordinated sub-step of chassis movement, two-arm grasping, and object placement; and the language instructions for the task. The language instructions for the task can be: "Control the wheeled mobile robot to move to the work point, place the red cube, blue cylinder, and green sphere on the table into the corresponding colored boxes, keep the objects stable throughout the process without tipping or slipping, and complete the grasping, handling, and storage through chassis alignment and coordinated two-arm operation." Step 202: Perform task semantic parsing on the language instructions to obtain task element information.

[0094] Step 203: Based on task element information, construct a task-level semantic graph. The task-level semantic graph includes root task nodes, sub-task nodes, atomic action nodes, node attributes, and relationships between nodes.

[0095] In this embodiment, hierarchical task semantic parsing is performed on language instructions to obtain task element information. Based on the task element information, root task nodes, sub-task nodes, atomic action nodes, node attributes, and relationships between nodes are obtained to construct a task hierarchical semantic graph.

[0096] For example, the language instructions for the collaborative task of the wheeled dual-arm robot can be hierarchically decomposed, and the root task node, sub-task node, atomic action node, node attributes, and relationships between nodes can be obtained by combining the wheeled chassis movement alignment dimension and the dual-arm collaboration constraint dimension.

[0097] For example, the root task is to move irregularly shaped objects on the desktop using wheels and to organize and store them using both arms.

[0098] Sub-task: (1) Alignment of wheeled chassis + coordinated arrangement of red blocks by both arms; (2) Wheel chassis alignment + double arm coordinated arrangement of slender blue cylinder; (3) Wheel chassis alignment + double arm coordinated sorting of irregular green spheres.

[0099] Atomic motion: Taking the collaborative sorting of large red squares by a wheeled dual-arm robot as an example, it includes: (1) The robot head global camera completes the overall scanning of the desktop scene, identifies the category of the red square, the global position, the position of the storage box and the distribution of obstacles on the desktop, and plans the optimal movement path of the chassis; (2) Control the wheeled chassis to move at low speed to the optimal working position directly in front of the red square, complete the fixed-point parking and attitude calibration, and eliminate the alignment deviation; (3) Turn on the left and right wrist cameras for close-range imaging, accurately detect the edge contour of the red square, the clamping contact surface and the offset angle, and determine the optimal clamping point on both sides; (4) The left and right robotic arms move synchronously to the preset clamping positions on both sides of the red square; (5) The dual end effectors work together to adjust the posture to ensure that the clamping positions on both sides are symmetrical and the force is balanced; (6) Both arms descend synchronously to the cooperative grasping height, maintaining horizontal alignment; (7) The left and right grippers close slowly and synchronously to collect the clamping torque and complete the stable clamping; (8) The target object is lifted synchronously and at a constant speed with both arms. The posture is corrected in real time by combining global and wrist vision to prevent the object from shifting or tipping over. (9) Based on the position of the red storage box, control the wheeled chassis to make fine adjustments to the alignment, and move it to the working area directly above the red box in coordination with the posture adjustment of the arms; (10) Both arms descend simultaneously to the placement height, matching the opening posture of the storage box; (11) The left and right grippers open and close simultaneously to release the object, completing the stable placement without damage; (12) Visually verify the posture and position of the red square using the head global camera to determine whether it remains stably inside the red box; (13) The arms are reset to the initial standby posture, the chassis returns to the standby position, and the single-movement collaborative operation is completed.

[0100] Step 204: Based on the task execution time intervals corresponding to the root task node, the task execution time intervals corresponding to the sub-task nodes, and the task execution time intervals corresponding to the atomic action nodes, determine the task execution segments corresponding to the root task node, the task execution segments corresponding to the sub-task nodes, and the task execution segments corresponding to the atomic action nodes, respectively.

[0101] Step 205: Based on visual data, robot state data, action sequences, task execution segments corresponding to the root task node, task execution segments corresponding to the sub-task nodes, and task execution segments corresponding to the atomic action nodes, extract the task start state data, segment action sequence, and task end state data of the root task node, the task start state data, segment action sequence, and task end state data of the sub-task nodes, and the task start state data, segment action sequence, and task end state data of the atomic action nodes.

[0102] In this embodiment, the task start state data, segment action sequence and task termination state data corresponding to the task execution segment of each task node can be identified from visual data, robot state data and action sequence.

[0103] For example, the task execution segment of the "chassis alignment with target" task node starts with the chassis starting to move and ends with the completion of parking calibration; the task execution segment of the "dual-arm collaborative grasping" task node starts with the dual arms reaching the gripping position and ends with the grippers holding the object stably and lifting it.

[0104] Step 206: Determine the task type of the root task node, the task type of the sub-task node, and the task type of the atomic action node respectively. Based on the preset correspondence between task type and criteria, obtain the task completion criteria of the root task node, the task completion criteria of the sub-task node, and the task completion criteria of the atomic action node respectively.

[0105] In this embodiment, the task start state data, fragment action sequence, and task end state data of each task node are combined with the task completion criterion of that task node to obtain the task start state data, fragment action sequence, task end state data, and task completion criterion of each task node.

[0106] Step 207: Based on the root task node, sub-task node, and atomic action node, obtain the root task-level language instructions, sub-task-level language instructions, and action-level language instructions.

[0107] Step 208: Perform semantic transformation processing on the root task-level language instructions, subtask-level language instructions, and action-level language instructions respectively, and verify the transformation results of the three types of language instructions respectively to obtain root task-level language instruction variants, subtask-level language instruction variants, and action-level language instruction variants.

[0108] In this embodiment, for example, when the dual-arm wheeled mobile robot performs the task of sorting and collaboratively transporting irregularly shaped objects on a desktop, the root task-level language instruction can be: "Control the dual-arm wheeled robot to complete the collaborative sorting and storage of irregularly shaped objects on the desktop."; the sub-task-level language instruction can be: "Control the wheeled chassis to align with the target point, and have the two arms work together to smoothly place the red cube into the red box, maintaining the stability of the object's posture."; the action-level language instruction can be: "Use the head camera to globally locate the red cube and plan the chassis's movement path."; "Control the wheeled robot to move to the optimal working position in front of the red cube." "And calibrate while parked."; "Activate the left and right wrist cameras for close-range alignment, and fine-tune the gripping posture of the dual robotic arms' ends."; "Lower both arms synchronously to the collaborative gripping height, maintaining horizontal alignment."; "The left and right grippers close slowly and synchronously, relying on wrist visual feedback to complete stable gripping."; "Both arms raise the red cube synchronously, combining global and local visual feedback to correct the posture in real time."; "Fine-tune the position of the wheeled chassis, coordinating with the movement of the two arms to directly above the red box."; "Lower the two arms to the placement height, and fine-tune the alignment accuracy using the wrist cameras."; "Open the grippers to release the red cube, completing stable placement." Furthermore, taking the subtask-level language instruction "Control the wheeled chassis to align with the target point, and coordinate both arms to smoothly place the red cube into the red box, maintaining the object's stable posture" as an example, variations of this subtask-level language instruction may include: Perspective variation: "Control the wheeled dual-arm robot to move to the work area, and use the left and right arms to grip the red cube and smoothly transfer it into the red box." Result state variant: "Makes the red square on the table finally stable inside the red box, without tipping over or shifting." Abstract level variant: "Complete the autonomous alignment, grasping, and repositioning of the red irregular object on the desktop." A specific variation of the description reads: "Using the head camera for global positioning and the wrist camera for precise alignment, the wheeled robot is controlled to move to the target position, and its two arms symmetrically grip the red cube on the table, smoothly transferring and placing it in the center of the red box." Spatial constraint variant: "Smoothly move the red cube to the center area inside the red box, avoiding collisions with the tabletop, box walls, and surrounding obstacles during the operation." A variation of the timing description is: "First, the chassis path is planned through visual positioning, the target is moved and aligned, then the two arms work together to grab the object, and finally it is moved to the storage area to complete the placement and reset." Step 209: Based on root task-level language instructions, subtask-level language instructions, action-level language instructions, root task-level language instruction variants, subtask-level language instruction variants, and action-level language instruction variants, obtain root task-level language instruction enhancement data, subtask-level language instruction enhancement data, and action-level language instruction enhancement data, respectively.

[0109] Step 210: Based on the task start state data, segment action sequence, task termination state data, and task completion criteria of the root task node, sub-task node, and atomic action node, as well as the root task-level language instruction enhancement data, sub-task-level language instruction enhancement data, and action-level language instruction enhancement data, obtain training sample data, and train the visual language action model to be trained based on the training sample data.

[0110] In this embodiment, firstly, based on the training sample data, the root task-level sample data, sub-task-level sample data, and action-level sample data can be selected according to a preset selection ratio. Secondly, based on the result of the selection process, the visual language action model to be trained can be trained to obtain training evaluation information. Thirdly, in response to the training evaluation information indicating a high-risk situation, the preset selection ratio can be adjusted. Finally, based on the training sample data and according to the result of the adjustment process, the root task-level sample data, sub-task-level sample data, and action-level sample data can be updated.

[0111] For example, the action task-level language instructions related to the subtask-level language instruction "Control the wheeled chassis to align with the target point, and coordinate the two arms to smoothly place the red cube into the red box while maintaining the object's stable posture" could include: "Scan the scene using the head camera to accurately locate the coordinates of the red cube."; "Control the wheeled chassis to fine-tune its position and calibrate the robot's relative standing position to the target."; "Activate the left and right wrist cameras to closely calibrate the gripping points at the ends of the two arms."; "Simultaneously lower both arms to the standard gripping height."; "Close the left and right grippers and maintain a constant gripping torque to complete a stable gripping operation." Understandably, during model training, if the wheeled dual-arm robot fails to grasp the red square twice in a row, an adaptive instruction degradation mechanism can be triggered. This mechanism reduces the proportion of subtask-level language instructions in the training sample data and increases the proportion of action-level task-level language instructions in the training sample data, thereby updating the training sample data and retraining the model.

[0112] Figure 3 This is a schematic diagram of the architecture of a data augmentation method provided in an embodiment of this application, as shown below. Figure 3As shown, firstly, teaching data acquisition is performed. The raw teaching data can include 30 frames / second of global head images, left and right wrist RGB images, joint position, velocity, and torque data of both arms, six-dimensional pose of the end effector, relative posture data of both arms, gripper opening and closing and clamping torque data, wheeled chassis movement speed, displacement, and heading angle data, as well as deviation data for each sub-step and task success / failure labels, realizing multi-dimensional data and language commands for vision, chassis, arms, and gripping. Secondly, data preprocessing is performed. To address the issues of asynchronous acquisition by three cameras and inconsistent timing of chassis and arm movements, a unified timestamp benchmark is used to complete missing frame completion, motion anomaly filtering, and multi-sensor spatial alignment. This corrects the perspective deviation between global vision and local wrist vision, and the timing deviation between chassis movement and arm actions, outputting standardized teaching data with unified timing and regular dimensions. Standardized teaching data can include visual images, robot motion information, and language commands. Robot motion information can include robot state data and action sequences. Thirdly, visual state feature extraction processing can be performed on the visual images. The robot's motion information is analyzed for state and action. Semantic parsing of language commands is performed, a task-level semantic graph is constructed, and task nodes are managed to obtain multiple task nodes, including root task nodes, sub-task nodes, and action task nodes. Next, the results of visual state feature extraction, state and action parsing of robot motion information, and execution fragment identification of multiple task nodes are performed. Then, based on the identified execution fragments, command-state-action alignment is performed. Next, task completion criteria are generated for each task node. Then, for the language commands of each task node, multi-granularity command generation, multi-dimensional language enhancement, and semantic consistency verification are performed first, filtering out invalid variants and retaining high-quality enhanced command samples. Finally, the aligned command-state-action data is associated one-to-one with the verified multi-dimensional language command enhancement data to construct training sample data. The VLA model is trained or fine-tuned based on this training sample data, enabling the model to learn the mapping relationship between different language expressions, different command granularities, multi-view visual states, and wheeled dual-arm compound actions.

[0113] Thus, by adopting the scheme in this embodiment, the complete robot task language instructions can be decomposed into a three-level structure of root task, sub-task, and atomic action. At the same time, by combining the initial state data, fragment action sequence, task termination state data, completion criteria, and other information in the task semantic graph, enhanced model training sample data can be obtained. This enables the VLA model to accurately distinguish between "global tasks", "single-object collaborative sub-tasks", and "chassis alignment / fine grasping / precise placement atomic actions", significantly improving the VLA model's task understanding accuracy for complex mobile operations.

[0114] Furthermore, by adopting the scheme in this embodiment, the VLA model can fully learn the relationship between natural language and multi-mechanism compound actions through enhanced model training sample data, effectively improving the model's ability to adapt to user-differentiated language commands and reducing the standardization requirements of human-computer interaction commands.

[0115] Furthermore, by employing the solution in this embodiment, the initial state data, segment action sequences, task termination state data, completion criteria, and corresponding level task language instructions can be combined to achieve a complete closed-loop data matching of "pre-task state - executed action - post-task state" for each language instruction. The model not only learns "what action to execute," but also accurately learns "in what scene location and robot posture to execute the action" and "the standard state required for action completion." Based on the trained VLA model, robot motion generation can solve problems such as perception and motion misalignment, inaccurate alignment, and gripping offset in mobile dual-arm robots.

[0116] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0117] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0118] Figure 4 A structural block diagram of a data enhancement apparatus according to an embodiment of this application is shown, as follows: Figure 3As shown. The data augmentation device 400 of this embodiment may include an acquisition unit 401, a construction unit 402, an alignment unit 403, an enhancement unit 404, a determination unit 405, and a training unit 406. The acquisition unit 401 is used to acquire teaching data to be processed, the teaching data including visual data, robot state data, action sequences, and language instructions corresponding to the action sequences; the construction unit 402 is used to construct a task-level semantic graph based on the language instructions, the task-level semantic graph including root task nodes, sub-task nodes, and atomic action nodes; the alignment unit 403 is used to obtain, based on the visual data, robot state data, and action sequences, and using a preset alignment strategy, the task start state data, fragment action sequences, task termination state data, and task completion criteria of the root task node, sub-task node, and atomic action node, respectively; the enhancement unit 404 is used to... Based on the root task node, sub-task node, and atomic action node, and using a preset language instruction enhancement strategy, root task-level language instruction enhancement data, sub-task-level language instruction enhancement data, and action-level language instruction enhancement data are obtained respectively; the determination unit 405 is used to determine training sample data based on the task start state data, segment action sequence, task termination state data, and task completion criteria of the root task node, sub-task node, and atomic action node, as well as the root task-level language instruction enhancement data, sub-task-level language instruction enhancement data, and action-level language instruction enhancement data; the training unit 406 is used to train the visual language action model to be trained based on the training sample data.

[0119] Optionally, in one possible implementation of this embodiment, the construction unit 402 is used to perform task semantic parsing processing on the language instructions to obtain task element information; based on the task element information, to generate root task nodes, sub-task nodes, atomic action nodes, and the relationships between nodes; and based on the root task nodes, sub-task nodes, atomic action nodes, and the relationships between nodes, to construct a task hierarchical semantic graph.

[0120] Optionally, in one possible implementation of this embodiment, the alignment unit 403 is used to determine the task execution segments corresponding to the root task node, sub-task node, and atomic action node based on the task execution time intervals corresponding to the root task node, sub-task node, and atomic action node, respectively; and to extract the task start state data, segment action sequence, and task termination state data of each task execution segment based on the visual data, robot state data, and action sequence, respectively. The task types of the root task node, sub-task node, and atomic action node are determined respectively; based on the preset correspondence between task types and criteria, the task completion criteria of the root task node, sub-task node, and atomic action node are obtained respectively. Optionally, in one possible implementation of this embodiment, the enhancement unit 404 is configured to: obtain root task-level language instructions, sub-task-level language instructions, and action-level language instructions based on the root task node, sub-task node, and atomic action node; perform semantic transformation processing on the root task-level language instructions, sub-task-level language instructions, and action-level language instructions respectively to obtain candidate root task-level language instruction variants, candidate sub-task-level language instruction variants, and candidate action-level language instruction variants; perform verification processing on the root candidate task-level language instruction variants, candidate sub-task-level language instruction variants, and candidate action-level language instruction variants respectively based on the task-level language instructions, sub-task-level language instructions, and action-level language instructions to obtain root task-level language instruction variants, sub-task-level language instruction variants, and action-level language instruction variants respectively; and obtain root task-level language instruction enhancement data, sub-task-level language instruction enhancement data, and action-level language instruction enhancement data respectively based on the root task-level language instructions, sub-task-level language instructions, action-level language instructions, root task-level language instruction variants, sub-task-level language instruction variants, and action-level language instruction variants.

[0121] Optionally, in one possible implementation of this embodiment, the determining unit 405 is used to obtain root task-level sample data based on root task-level language instruction enhancement data and the start state data, fragment action sequence, and task termination state data of the root task node; to obtain sub-task-level sample data based on sub-task-level language instruction enhancement data and the start state data, fragment action sequence, and task termination state data of the sub-task node; to obtain action-level sample data based on action-level language instruction enhancement data and the start state data, fragment action sequence, and task termination state data of the atomic action node; and to obtain training sample data based on the root task-level sample data, sub-task-level sample data, and action-level sample data.

[0122] Optionally, in one possible implementation of this embodiment, the training unit 406 is configured to: select the root task-level sample data, sub-task-level sample data, and action-level sample data according to a preset selection ratio based on the training sample data; train the visual language action model to be trained based on the result of the selection process to obtain training evaluation information; adjust the preset selection ratio in response to the training evaluation information indicating a high-risk situation; and update the selection of the root task-level sample data, sub-task-level sample data, and action-level sample data according to the result of the adjustment process based on the training sample data, so as to train the visual language action model to be trained based on the result of the updated selection process.

[0123] In this embodiment, the teaching data to be processed can be acquired by the acquisition unit. The teaching data includes visual data, robot state data, action sequences, and language instructions corresponding to the action sequences. Then, the construction unit can construct a task-level semantic graph based on the language instructions. The task-level semantic graph includes root task nodes, sub-task nodes, and atomic action nodes. The alignment unit, based on the visual data, robot state data, and action sequences, uses a preset alignment strategy to obtain the task start state data, fragment action sequences, task termination state data, and task completion criteria of the root task nodes, sub-task nodes, and atomic action nodes, respectively. The enhancement unit, based on the root task nodes, sub-task nodes, and atomic action nodes, uses a preset language instruction enhancement strategy to obtain root task-level language instruction enhancement data, sub-task-level language instruction enhancement data, and action-level language instruction enhancement data, respectively. The determination unit, based on the task start state of the root task nodes, sub-task nodes, and atomic action nodes, uses a preset language instruction enhancement strategy to obtain root task-level language instruction enhancement data, sub-task-level language instruction enhancement data, and action-level language instruction enhancement data, respectively. The training sample data consists of data such as fragment action sequences, task termination state data, and task completion criteria, as well as root task-level language instruction enhancement data, sub-task-level language instruction enhancement data, and action-level language instruction enhancement data. The training unit then uses this training sample data to train the visual language action model to be trained. Since the training sample data can be combined with the robot's initial state, action sequence, termination state, and task completion criteria during robot execution to generate multiple language instruction enhancement data with different levels of granularity without significantly increasing the cost of collecting real robot teaching data, and thus automatically constructing a five-element training sample data structure of multi-granularity instruction—initial state—action sequence—termination state—completion criterion, this data is used to train the visual language action model. This improves the visual language action model's language understanding ability, state-action correspondence ability, and task execution stability for complex embodied tasks.

[0124] The technical solution of this application involves the collection, storage, use, processing, transmission, provision, and disclosure of user personal information, such as user image and attribute data, which comply with relevant laws and regulations and do not violate public order and good morals.

[0125] According to embodiments of this application, this application also provides an electronic device, a readable storage medium, and a computer program product.

[0126] Figure 5A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0127] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data required for the operation of the electronic device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0128] Multiple components in electronic device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows electronic device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0129] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as data augmentation methods. For example, in some embodiments, the data augmentation method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the data augmentation method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform data augmentation methods by any other suitable means (e.g., by means of firmware).

[0130] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0131] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0132] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0133] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0134] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0135] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0136] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0137] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method of data augmentation, characterized in that, The method includes: Acquire teaching data to be processed, the teaching data including visual data, robot state data, action sequences, and language instructions corresponding to the action sequences; Based on the language instructions, a task-level semantic graph is constructed, which includes a root task node, sub-task nodes, and atomic action nodes. Based on the visual data, robot state data, and action sequence, and using a preset alignment strategy, the task start state data, segment action sequence, task termination state data, and task completion criteria of the root task node, sub-task node, and atomic action node are obtained respectively. Based on the root task node, sub-task node, and atomic action node, and using a preset language instruction enhancement strategy, root task-level language instruction enhancement data, sub-task-level language instruction enhancement data, and action-level language instruction enhancement data are obtained respectively. Based on the task start state data, fragment action sequence, task termination state data and task completion criteria of the root task node, sub-task node, and atomic action node, as well as the root task-level language instruction enhancement data, sub-task-level language instruction enhancement data and action-level language instruction enhancement data, the training sample data is determined. Based on the training sample data, the visual language action model to be trained is trained.

2. The method of claim 1, wherein, The construction of a task-level semantic graph based on the language instructions includes: The language instructions are subjected to task semantic parsing to obtain task element information; Based on the task element information, a root task node, sub-task nodes, atomic action nodes, and the relationships between nodes are generated. Based on the root task node, sub-task nodes, atomic action nodes, and the relationships between nodes, a task-level semantic graph is constructed.

3. The method according to claim 1, characterized in that, Based on the visual data, robot state data, and action sequence, and using a preset alignment strategy, the task start state data, segment action sequence, task termination state data, and task completion criteria for the root task node, sub-task nodes, and atomic action nodes are obtained, respectively, including: Based on the task execution time intervals corresponding to the root task node, sub-task node, and atomic action node, the task execution segments corresponding to the root task node, sub-task node, and atomic action node are determined respectively. Based on the visual data, robot state data, and action sequence, the task start state data, segment action sequence, and task end state data of each task execution segment are extracted respectively. Determine the task types of the root task node, sub-task nodes, and atomic action nodes respectively; Based on the preset correspondence between task types and criteria, the task completion criteria of the root task node, sub-task node, and atomic action node are obtained respectively.

4. The method according to claim 1, characterized in that, Based on the root task node, sub-task nodes, and atomic action nodes, and using a preset language instruction enhancement strategy, root task-level language instruction enhancement data, sub-task-level language instruction enhancement data, and action-level language instruction enhancement data are obtained respectively, including: Based on the root task node, sub-task node, and atomic action node, obtain root task-level language instructions, sub-task-level language instructions, and action-level language instructions. Semantic transformation processing is performed on the root task-level language instructions, subtask-level language instructions, and action-level language instructions respectively to obtain candidate root task-level language instruction variants, candidate subtask-level language instruction variants, and candidate action-level language instruction variants. Based on the task-level language instructions, subtask-level language instructions, and action-level language instructions, the root candidate task-level language instruction variants, candidate subtask-level language instruction variants, and candidate action-level language instruction variants are respectively verified to obtain the root task-level language instruction variants, subtask-level language instruction variants, and action-level language instruction variants. Based on root task-level language instructions, subtask-level language instructions, action-level language instructions, root task-level language instruction variants, subtask-level language instruction variants, and action-level language instruction variants, respectively, root task-level language instruction enhancement data, subtask-level language instruction enhancement data, and action-level language instruction enhancement data are obtained.

5. The method according to claim 1, characterized in that, The training sample data, determined based on the task start state data, segment action sequence, task termination state data, and task completion criteria of the root task node, sub-task node, and atomic action node, as well as the root task-level language instruction enhancement data, sub-task-level language instruction enhancement data, and action-level language instruction enhancement data, includes: Based on the root task-level language instruction enhancement data and the root task node's start state data, fragment action sequence, and task termination state data, root task-level sample data is obtained. Based on the subtask-level language instruction enhancement data and the subtask node's start state data, fragment action sequence, and task termination state data, subtask-level sample data is obtained. Based on the action-level language instruction enhancement data and the start state data, fragment action sequence, and task termination state data of the atomic action nodes, action-level sample data is obtained. Training sample data is obtained based on the root task-level sample data, sub-task-level sample data, and action-level sample data.

6. The method according to claim 5, characterized in that, The training process based on the training sample data for the visual language action model to be trained includes: Based on the training sample data, the root task-level sample data, sub-task-level sample data, and action-level sample data are selected and processed according to a preset selection ratio. Based on the results of the selection process, the visual language action model to be trained is trained to obtain training evaluation information. In response to the training evaluation information indicating a high-risk situation, the preset selection ratio is adjusted. Based on the training sample data, and according to the results of the adjustment process, the root task-level sample data, sub-task-level sample data, and action-level sample data are updated and selected, so as to train the visual language action model to be trained based on the results of the update and selection process.

7. A data augmentation apparatus, characterized in that, The device includes: The acquisition unit is used to acquire teaching data to be processed, the teaching data including visual data, robot state data, action sequence, and language instructions corresponding to the action sequence; The construction unit is used to construct a task-level semantic graph based on the language instructions. The task-level semantic graph includes a root task node, sub-task nodes, and atomic action nodes. The alignment unit is used to obtain the task start state data, fragment action sequence, task termination state data and task completion criteria of the root task node, sub-task node and atomic action node respectively based on the visual data, robot state data and action sequence and using a preset alignment strategy. The enhancement unit is used to obtain root task-level language instruction enhancement data, sub-task-level language instruction enhancement data, and action-level language instruction enhancement data respectively based on the root task node, sub-task node, and atomic action node, using a preset language instruction enhancement strategy. The determining unit is used to determine training sample data based on the task start state data, fragment action sequence, task termination state data and task completion criteria of the root task node, sub-task node, and atomic action node, as well as the root task-level language instruction enhancement data, sub-task-level language instruction enhancement data and action-level language instruction enhancement data. The training unit is used to train the visual language action model to be trained based on the training sample data.

8. An electronic device, characterized in that, include: At least one processor; as well as A memory that is communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images