Target driving method and system for robot behavior learning

Abstract

The invention discloses a target driving method and system for robot behavior learning, and relates to behavior learning and artificial intelligence technologies, and the method comprises the steps: obtaining Play data, and expanding the obtained Play data based on a target state, any one of the Play data comprising paired tuples (ot, alpha t), ot representing observation at a t moment, and alpha t representing an action at the t moment; and based on the extended Play data, using a Goal-Driven Transformer GDT model to output an action probability distribution for target driving, the GDT model comprising an embedded layer, a target driving attention module and a standard attention module. A target-driven attention mechanism is provided, and the ability of the robot to identify and execute a complex action sequence to achieve an expected future or target state is significantly improved.

Description

Technical Field

[0001] The present application relates to the fields of behavior learning and artificial intelligence technology, and in particular to a target-driven method and system for robot behavior learning. Background Art

[0002] Robot behavior learning is a key area in robotics that involves training robots to perform various tasks autonomously through learning from experience. This field has significant potential in improving robot autonomy, enabling robots to perform tasks in unpredictable environments and adapt to new tasks without being explicitly programmed. As robotic systems become more complex and are expected to operate in increasingly dynamic environments, the importance of efficient robot behavior learning techniques becomes more apparent than ever.

[0003] One approach to learning robot behaviors is through offline reinforcement learning, which learns decisions by optimizing cumulative rewards from previous interactions in a fixed dataset. However, this approach relies heavily on the availability of reward labels, which are often difficult to specify accurately for complex tasks. In addition, the reliance on predefined reward functions can limit the flexibility and generality of learned behaviors, making it difficult to adapt to tasks where the reward structure is unclear or difficult to encode.

[0004] Another approach is imitation learning, where robots learn behaviors by imitating expert demonstrations. This approach bypasses the need for an explicit reward function by directly learning actions demonstrated by experts in similar situations. While imitation learning can effectively transfer expert knowledge to robots, it requires high-quality demonstrations, which can be expensive and time-consuming. Furthermore, this approach may perform poorly on tasks that are not easily demonstrated or where expert knowledge is not readily available.

[0005] In contrast, Play data contains unstructured and diverse interactions in the environment, providing an opportunity to overcome these limitations. Play data allows for richer exploration of behaviors, providing a broader scope for learning relative to structured expert demonstrations.

[0006] Recent work has explored the use of Play data in robot behavior learning, addressing the limitations of offline reinforcement learning and traditional imitation learning. For example, Shafiullah et al. introduced the Behavior Transformer (BeT), which integrates action discretization and correction into the standard transformer to predict multimodal continuous actions. However, the standard BeT is limited to unconditional behavior playback and cannot select specific behavior modes during policy deployment. Summary of the invention

[0007] The embodiments of the present application provide a goal-driven method and system for robot behavior learning, which is used to propose a goal-driven attention mechanism to significantly improve the robot's ability to recognize and execute complex action sequences to achieve an expected future or target state.

[0008] The present application embodiment provides a target-driven method for robot behavior learning, including:

[0009] Get Play data, and expand the acquired Play data based on the target state, wherein any Play data contains a pair of tuples (ο t ,α t ), t represents the observation at time t, α t represents the action at time t;

[0010] Based on the extended Play data, the Goal-Driven Transformer (GDT) model is used to output the action probability distribution for goal-driven action, where the GDT model includes an embedding layer, a goal-driven attention module, and a standard attention module;

[0011] The embedding layer is used to encode the input observation sequence, the corresponding time step sequence and the target state into observation embedding, position embedding and target embedding respectively;

[0012] the target-driven attention module, for processing the observation embedding, the position embedding and the target embedding encoded by the embedding layer to aggregate features and context information related to the output target through the target-driven attention block;

[0013] The standard attention module is used to optimize the understanding of the contextual relationship between observations and output targets.

[0014] Optionally, the Play data acquired based on the target state expansion includes: based on the target state Expanded pair tuple (ο t ,α t ) is a triple

[0015] Optionally, the embedding layer includes an observation embedding layer, a position embedding layer and a target embedding layer; wherein,

[0016] The observation embedding layer is used to encode the information contained in the observation sequence to convert the discrete observation sequence into a continuous vector representation;

[0017] The position embedding layer is used to provide position information and add position encoding to the tag embedding;

[0018] The target embedding layer is used to obtain the target of the state and convert it into a target embedding vector representation represented by a continuous vector.

[0019] Optionally, the target driven attention module is specifically used for:

[0020] The vector representation E converted by the target embedding layer g , using the linear layer to generate the query Q with attention mechanism g , Q g =Linear(E g );as well as,

[0021] Apply two linear layers to the observation embedding and position embedding and generate the key K of the attention mechanism op Sum value V op :

[0022] Q g =Linear(E g )

[0023] V op =Linear(E o +E p )

[0024] K op =Linear(E o +E p )

[0025] Compute the attention score using the scaled dot product attention mechanism

[0026]

[0027] Optionally, the target driven attention module is further used to:

[0028] Pay attention to the score Introducing value V op , to calculate the target driving context vector as well as,

[0029] Set up a skip connection to represent the target driving context vector and the vector output of the observation embedding layer E o The vector representation E output by the position embedding layer o The concatenation of , and , the vector representation E output by the target embedding layer g The vector representation E of the observation embedding layer output is the input embedding o The vector representation E output by the position embedding layer o , to perform splicing:

[0030]

[0031] Optionally, in the process of training the GDT model, an action discretization module is introduced to divide the real action A into discrete and continuous components:

[0032] Use the k-means clustering algorithm to determine k specified cluster categories to generate a set of centroids based on the determined cluster categories in Representing the action space In the training and evaluation process of the GDT model, the configuration of the k-means encoder and decoder remain consistent;

[0033] During the training phase of GDT, the k-means encoder transforms the real action Divided into:

[0034]

[0035]

[0036] in, represents the discrete action centroid that is closest to the real action, A continuous residual component representing an action.

[0037] Optionally, the predictions generated by the GDT model include discrete Continuous with action

[0038] Set the following loss function:

[0039]

[0040] Among them, L focal is the Focal loss based on negative log-likelihood, L MT It is called Masked Multi-Task loss.

[0041] An embodiment of the present application further proposes a target-driven system for robot behavior learning, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the target-driven method for robot behavior learning as described above are implemented.

[0042] The embodiments of the present application propose a goal-driven attention mechanism, which significantly improves the robot's ability to recognize and execute complex action sequences to achieve an expected future or target state.

[0043] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0045] Figure 1 This is a basic flow diagram of the target-driven method for robot behavior learning in an embodiment of the present application;

[0046] Figure 2 The structure of the GDT model of the embodiment of the present application is shown in FIG.

[0047] Figure 3 The schematic diagram of the architecture of the target driven attention module of the embodiment of the present application;

[0048] Figure 4 This is a schematic diagram of the result of completing the block pushing task in the application example of this application;

[0049] Figure 5 This is an illustration of the kitchen task completion results for this application example. DETAILED DESCRIPTION

[0050] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0051] The present application embodiment provides a target-driven method for robot behavior learning, such as Figure 1 As shown, the following steps are included:

[0052] In step S101, Play data is obtained, and the obtained Play data is expanded based on the target state, wherein any Play data contains a pair of tuples ( t ,α t ), the data set itself is (O, A, G), t represents the observation at time t, α t Represents the action at time t.

[0053] In some embodiments, the Play data acquired based on the target state expansion includes: based on the target state Expanded pair tuple (ο t ,α t ) is a triple As a concrete example, consider a tuple containing pairs Play dataset, this application uses triples To expand the data set, add the target state to each pair of original tuples, thereby enhancing the model pair with the target state The goal of this application is to learn a strategy This strategy is able to model the t Transition to future target state The action distribution of the behavior model is to optimize the strategy, and its formula is:

[0054]

[0055] In the embodiment of the present application, a goal-driven transformer is proposed to promote the behavior learning process by focusing on achieving a specific goal state from the Play data. Specifically, in step S102, based on the extended Play data, the GDT model is used to output the action probability distribution for goal driving. Figure 2 The architecture of the GDT model is shown, where the GDT model includes an embedding layer, a target-driven attention module, and a standard attention module.

[0056] The embedding layer is used to encode the input observation sequence, the corresponding time step sequence and the target state into observation embedding, position embedding and target embedding respectively;

[0057] the target-driven attention module, for processing the observation embedding, the position embedding and the target embedding encoded by the embedding layer to aggregate features and context information related to the output target through the target-driven attention block;

[0058] The standard attention module is used to optimize the understanding of the contextual relationship between observations and output targets.

[0059] The embodiments of the present application propose a goal-driven attention mechanism, which significantly improves the robot's ability to recognize and execute complex action sequences to achieve an expected future or target state.

[0060] The embedding layer serves as the entry point of the transformer model and converts the input data into a continuous vector representation. In some embodiments, the embedding layer includes an observation embedding layer, a position embedding layer, and a target embedding layer, each of which is responsible for encoding different aspects of the input; wherein,

[0061] The observation embedding layer is used to encode the information in the multimodal observations. The observation embedding subcomponent processes the input data. It converts the labeled sequence into a continuous vector E o , effectively capturing the observed content.

[0062] The position embedding layer is used to provide position information to the model. The position embedding component adds position encoding E to the tag embedding. p ,These encodings enable the model to understand the relative order and position of tokens in the input sequence, preserving important spatial relationships. .

[0063] The target embedding layer is used to obtain the high-level target or target of the state and convert it into a target embedding E represented by a continuous vector g .

[0064] The goal embedding in the goal embedding layer refines the core goal into a form that serves as a directional guide for the model, ensuring that the processing is always consistent with achieving the predetermined goal state.

[0065] The above observation embedding layer, position embedding layer and target embedding layer are formulated as:

[0066] Q g =Linear(E g )

[0067] V op =Linear(E o +E p )

[0068] K op =Linear(E o +E p ).

[0069] The goal-driven attention module is a key component in the GDT model, and its strategic design gives the model the ability to accurately align high-level goals or objectives. The goal-driven attention module acts as a driving force for the model's goal-aware processing, enabling it to excel in tasks that require a deep understanding of task-specific goals and effective use of multimodal data. Figure 3 The architecture of the target driven attention module is shown. In some embodiments, the target driven attention module is specifically used to:

[0070] The vector representation E converted by the target embedding layer g , using the linear layer to generate the query Q with attention mechanism g , Q g =Linear(E g );as well as,

[0071] Apply two linear layers to the sum of the observation embedding and the position embedding to generate the key K of the attention mechanismop Sum value V op :

[0072] Q g =Linear(E g )

[0073] V op =Linear(E o +E p )

[0074] K op =Linear(E o +E p )

[0075] Compute the attention score using the scaled dot product attention mechanism

[0076]

[0077] In some embodiments, the target driven attention module is further used to:

[0078] Pay attention to the score Introducing value V op , to calculate the target driving context vector as well as,

[0079] Set up a skip connection to represent the target driving context vector and the vector output of the observation embedding layer E o The vector representation E output by the position embedding layer o The concatenation of , and , the vector representation E output by the target embedding layer g The vector representation E of the observation embedding layer output is the input embedding o The vector representation E output by the position embedding layer o , to perform splicing:

[0080]

[0081] Utilizing skip connections in this application allows the model to retain original contextual information while incorporating goal-driven insights, and setting up skip connections ensures that valuable information from early processing stages is retained.

[0082] By taking the target embedding as a query, applying an attention mechanism to the observation embedding, and adding skip connections, our target-driven attention module not only enhances the model’s attention to target-related information, but also maintains a rich contextual understanding of the input data, enabling the model to make intelligent decisions and actions based on both high-level task goals and underlying context.

[0083] In some embodiments, the process of training the GDT model further includes introducing an action discretization module to divide the real action A into discrete and continuous components:

[0084] Use the k-means clustering algorithm to determine k specified cluster categories to generate a set of centroids based on the determined cluster categories in Representing the action space In the training and evaluation process of the GDT model, the configuration of the k-means encoder and decoder remain consistent;

[0085] During the training phase of GDT, the k-means encoder transforms the real action Divided into:

[0086]

[0087]

[0088] in, represents the discrete action centroid that is closest to the real action, A continuous residual component representing an action.

[0089] In some embodiments, the predictions generated by the GDT model include discrete Continuous with action

[0090] Set the following loss function:

[0091]

[0092] Among them, Lfoca l is the Focal loss based on negative log-likelihood, L MT It is called Masked Multi-Task loss.

[0093] This application proposes a GDT model specifically for robot behavior learning from Play data. The GDT architecture consists of three key components: embedding layer, goal-driven attention block (GDAB) and standard attention block (SAB).

[0094] Among them, this application designs GDAB, which uses an attention mechanism to guide the model to focus on specific targets, so that GDT can simplify the decision-making process by emphasizing relevant data and ignoring irrelevant information.

[0095] Specifically, the GDT model first uses an embedding layer to convert raw data into a structured format suitable for neural network analysis. Subsequently, the model uses the combined GDAB and SAB to support the learning of robot actions. GDAB optimizes the model's focus on achieving set goals by emphasizing goal-related features, while SAB ensures that this goal-oriented learning is built on a comprehensive understanding of the surrounding environment.

[0096] The effectiveness of the GDT model proposed in this application has been fully verified in two simulation environments. In the simulation environment, the GDT model performs well and surpasses other state-of-the-art robot behavior learning models. Performance indicators are used in the specific verification examples of this application to evaluate the effectiveness of the algorithm. In the push block setting, success is measured by correctly placing the green and red blocks in the target squares. In the kitchen setting, the success rate is determined by the proportion of completing four conditional tasks. The GDT method is compared with the established methods, and the results are shown in Table I.

[0097] TABLEI

[0098] CBET RESULT COMPARISON

[0099]

[0100] In both simulated environments, our approach showed performance advantages. Starting with the block-pushing environment, our model achieved a 96% success rate, outperforming traditional methods. Comparative analysis with BeT shows that incorporating goal conditions improves performance metrics. This gain is further amplified by integrating goal-driven attention blocks, resulting in superior results compared to C-BeT. Shifting attention to the kitchen environment, our model achieved an average score of 3.05 (out of 4), outperforming existing methods. A similar performance improvement trend was observed here, confirming the effectiveness of our approach in different scenarios.

[0101] In order to comprehensively evaluate the algorithm proposed in this application, the applicant has expanded the experimental framework in other embodiments, including increasingly complex test scenarios. Table II and Table III show detailed and quantitative comparisons of the performance indicators of the algorithm of this application with a series of state-of-the-art benchmarks, including multi-layer perceptron and mean square error (RBC), nearest neighbor (NN), local weighted regression (LWR), variational autoencoder (VAE), regularized flow (Flow) and implicit behavior cloning (IBC).

[0102] TABLE II

[0103] RESULTFOR BLOCKPUSH

[0104]

[0105] TABLE III

[0106] RESULTFORKITCHEN

[0107] Kitchen

[0108]

[0109] To understand the push block task more clearly, Table II divides it into “reach” and “push” subtasks. R1 and R2 represent successful arrival at the block, while P1 and P2 represent successful positioning at the target, e.g. Figure 4 As shown, the method of the present application allows for detailed probability evaluation of each subtask, and the model of the present application performs well in tasks that require coordinated execution of reaching and positioning actions.

[0110] In the evaluation of the kitchen environment, this application goes deeper than typical global metrics. Table III not only considers general metrics, but also calculates the tasks that the robot successfully completed. This provides detailed insights into the versatility of the algorithm. Figure 5 The robot was shown completing tasks using a variety of appliances. Our model outperformed other leading algorithms, especially in long-view tasks involving four or five of the seven tasks. Our model's performance remained robust as the complexity of the tasks grew, demonstrating its effectiveness in challenging scenarios.

[0111] The transformer using the target-driven attention block in this application shows advantages over the transformer relying only on standard attention in many aspects. These benefits are particularly evident in the task of expanding the field of view, and the method of the embodiment of this application consistently outperforms conditional and unconditional behavior transformers in benchmarks. As the field continues to pursue models capable of more detailed and context-aware information processing, the inclusion of target-driven attention blocks may be the key to driving the development of the next wave of deep learning architectures.

[0112] An embodiment of the present application further proposes a target-driven system for robot behavior learning, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the target-driven method for robot behavior learning as described above are implemented.

[0113] It should be noted that in the various embodiments of the present application, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0114] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0115] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0116] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A goal-driven method for robot behavior learning, characterized in that: include: Get Play data, and expand the acquired Play data based on the target state, wherein any Play data contains a pair of tuples (ο t ,α t ), t represents the observation at time t, α t represents the action at time t; Based on the extended Play data, the Goal-Driven Transformer (GDT) model is used to output the action probability distribution for goal-driven action, where the GDT model includes an embedding layer, a goal-driven attention module, and a standard attention module; The embedding layer is used to encode the input observation sequence, the corresponding time step sequence and the target state into observation embedding, position embedding and target embedding respectively; the target-driven attention module for processing the observation embedding, the position embedding and the target embedding encoded by the embedding layer to aggregate features and context information related to the output target through the target-driven attention block; The standard attention module is used to optimize the understanding of the contextual relationship between observations and output targets.

2. The goal-driven method for robot behavior learning according to claim 1, characterized in that: The Play data obtained based on the target state expansion includes: Expanded pair tuple (ο t ,α t ) is a triple 3. The goal-driven method for robot behavior learning according to claim 1, characterized in that: The embedding layer includes an observation embedding layer, a position embedding layer and a target embedding layer; wherein, The observation embedding layer is used to encode the information contained in the observation sequence to convert the discrete observation sequence into a continuous vector representation; The position embedding layer is used to provide position information and add position encoding to the tag embedding; The target embedding layer is used to obtain the target of the state and convert it into a target embedding vector representation represented by a continuous vector.

4. The target-driven method for robot behavior learning as claimed in claim 3, characterized in that: The target driven attention module is specifically used for: The vector representation E converted by the target embedding layer g , using the linear layer to generate the query Q with attention mechanism g , Q g =Linear(E g );as well as, Apply two linear layers to the observation embedding and position embedding and generate the key K of the attention mechanism op Sum value V op : Q g =Linear(E g ) V op =Linear(E o +E p ) K op =Linear(E o +E p ) Compute the attention score using the scaled dot product attention mechanism 5. The goal-driven method for robot behavior learning as claimed in claim 3, characterized in that: The target driven attention module is further used for: Pay attention to the score Introducing value V op , to calculate the target driving context vector as well as, Set up a skip connection to represent the target driving context vector and the vector output of the observation embedding layer E o The vector representation E output by the position embedding layer o The concatenation of , and , the vector representation E output by the target embedding layer g The vector representation E of the observation embedding layer output is the input embedding o The vector representation E output by the position embedding layer o , to perform splicing:

6. The goal-driven method for robot behavior learning according to claim 5, characterized in that: It also includes introducing an action discretization module during the training of the GDT model to divide the real action A into discrete and continuous components: Use the k-means clustering algorithm to determine k specified cluster categories to generate a set of centroids based on the determined cluster categories in Representing the action space In the training and evaluation process of the GDT model, the configuration of the k-means encoder and decoder remain consistent; During the training phase of GDT, the k-means encoder transforms the real action Divided into: in, represents the discrete action centroid that is closest to the real action, A continuous residual component representing an action.

7. The goal-driven method for robot behavior learning according to claim 6, characterized in that: The forecasts generated by the GDT model include discrete Continuous with action Set the following loss function: Among them, L focal is the Focal loss based on negative log-likelihood, L MT It is called Masked Multi-Task loss.

8. A goal-driven system for robot behavior learning, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the target-driven method for robot behavior learning as described in any one of claims 1 to 7 are implemented.

Images