Multi-task prompt decision transformer construction method and device, equipment and storage medium

Abstract

A multi-task prompt decision transformer construction method and device, equipment and storage medium, relate to the field of robot control simulation environment, including constructing MPDT, MPDT includes multiple decision maker layers and triple prompt module constructed based on reward, state and action;Control target decision maker layer and triple prompt module are in active state, MPDT is trained based on offline training sample, and multi-task prompt fine-tuning is carried out based on triple prompt, specific task prompt and cross-task prompt are generated;After training is completed, the combination of cross-task prompt and test sample is used as the input of MPDT for testing, the hidden state test mean and hidden state test variance corresponding to the test sample are calculated, and the mean and variance of the offline training sample are aligned and calculated;Based on the alignment result, the cross-task prompt is updated, the target multi-task prompt decision transformer is generated, and the generalization ability of the transformer in the zero sample and multi-task scene is improved.

Description

Technical Field

[0001] This application relates to the field of robot control simulation environment technology, specifically to a method, apparatus, device, and storage medium for constructing a multi-task prompting decision converter. Background Technology

[0002] In intelligent robot research, multi-task learning and zero-shot learning are two important research directions that can help robots achieve more flexible and intelligent interaction and learning. Multi-task learning aims to improve the generalization ability of each task by solving multiple related tasks simultaneously through shared representation learning, while zero-shot learning focuses on how to enable the model to recognize, classify, or make decisions about samples or categories not seen during training. The powerful capabilities of the Transformer architecture have made it one of the mainstream architectures in natural language processing, as it can handle long-range dependencies and learn global semantic information of text. This makes the Transformer architecture perform well in various tasks, including robot control, machine translation, text summarization, and question answering.

[0003] In related technologies, the converter architecture used in robot control is often built based on multi-task learning. However, the training data and test data used in the multi-task learning process are exactly the same, which makes it impossible for the converter to recognize, classify or make decisions on samples or categories that have not been seen during the training process. In other words, the current converter has poor generalization ability in zero-shot scenarios. In other words, intelligent robots built based on the current converter are not suitable for zero-shot scenarios. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for constructing a multi-task cue decision transformer, so as to improve the generalization ability of the transformer in zero-shot and multi-task scenarios.

[0005] In a first aspect, embodiments of this application provide a method for constructing a multi-task prompting decision transformer, comprising the following steps:

[0006] A multi-task cueing decision transformer (MPDT) is constructed, which includes a multi-layer decision transformer layer and a triple cueing module based on reward, state, and action.

[0007] At least one target decision layer is randomly selected from the multi-layer decision layer, and the target decision layer and the triplet prompting module are kept in an active state. The MPDT is trained based on a preset offline training sample. During the training process, the MPDT is fine-tuned based on the triplet prompts to generate specific task prompts and cross-task prompts.

[0008] After the MPDT training is completed, the combination of the cross-task prompts and unlabeled test samples is used as the input of the MPDT for testing, and the hidden state test mean and hidden state test variance corresponding to the test samples are calculated.

[0009] Alignment loss is obtained by performing alignment calculation based on the hidden state test mean and hidden state test variance, as well as the hidden state training mean and hidden state training variance corresponding to the offline training samples.

[0010] Backpropagation of the MPDT is performed based on the alignment loss to update cross-task cues and generate a target multi-task cue decision transformer.

[0011] In conjunction with the first aspect, in one implementation, prior to the step of training the MPDT based on preset offline training samples, the method further includes:

[0012] The multi-layer decision maker is initialized based on the pre-trained language model GPT2-SMALL to determine the parameter weights in the multi-layer decision maker.

[0013] In conjunction with the first aspect, in one implementation, the specific task cue is composed of a first low-rank vector and a second low-rank vector, wherein the first low-rank vector is determined based on the cue length and a low-rank parameter, and the second low-rank vector is determined based on the cue dimension and the low-rank parameter.

[0014] In conjunction with the first aspect, in one implementation, calculating the hidden state test mean and hidden state test variance corresponding to the test sample includes:

[0015] For each hidden layer in the MPDT, the hidden state test mean is calculated based on the hidden state of each test sample in the hidden layer, and the hidden state test variance is calculated based on the hidden state test mean.

[0016] In conjunction with the first aspect, in one implementation, the step of calculating the alignment loss based on the hidden state test mean and hidden state test variance, and the hidden state training mean and hidden state training variance corresponding to the offline training samples, includes:

[0017] Substituting the hidden state test mean, hidden state test variance, hidden state training mean, and hidden state training variance for each hidden layer into the following formula, we obtain the alignment loss:

[0018]

[0019] In the formula, L align Let L represent the alignment loss, and L represent the total number of hidden layers. This represents the test mean of the hidden states of the l-th hidden layer. This represents the test variance of the hidden state in the l-th hidden layer. This represents the training mean of the hidden states of the l-th hidden layer. Let represent the training variance of the hidden state of the l-th hidden layer.

[0020] Secondly, embodiments of this application provide a multi-task prompting decision transformer construction device, which includes: a construction module, a training module, and a testing module;

[0021] The building module is used to build a task prompting decision transformer (MPDT), which includes a multi-layer decision transformer layer and a triple prompting module based on reward, state, and action.

[0022] The training module is used to randomly select at least one target decision layer from the multi-layer decision layer, control the target decision layer and the triplet prompting module to be in an active state, and train the MPDT based on preset offline training samples. During the training process, the MPDT is fine-tuned for multi-task prompting based on triplet prompts to generate specific task prompts and cross-task prompts.

[0023] The testing module is used to test the combination of the cross-task prompts and unlabeled test samples as input to the MPDT after the MPDT training is completed, and to calculate the hidden state test mean and hidden state test variance corresponding to the test samples; and to perform alignment calculation based on the hidden state test mean and hidden state test variance and the hidden state training mean and hidden state training variance corresponding to the offline training samples to obtain the alignment loss.

[0024] The building module is also used to backpropagate the MPDT based on the alignment loss to achieve cross-task cue updates and generate a target multi-task cue decision transformer.

[0025] In conjunction with the second aspect, in one implementation, the building module is further configured to: initialize the multi-layer decision maker layer based on the pre-trained language model GPT2-SMALL, so as to determine the parameter weights of the multi-layer decision maker layer.

[0026] In conjunction with the second aspect, in one implementation, the specific task cue is composed of a first low-rank vector and a second low-rank vector, wherein the first low-rank vector is determined based on the cue length and a low-rank parameter, and the second low-rank vector is determined based on the cue dimension and the low-rank parameter.

[0027] In conjunction with the second aspect, in one implementation, the test module is specifically used to: for each hidden layer in the MPDT, calculate the hidden state test mean based on the hidden state of each test sample in the hidden layer, and calculate the hidden state test variance based on the hidden state test mean.

[0028] In conjunction with the second aspect, in one implementation, the testing module is further configured to: substitute the hidden state test mean, hidden state test variance, hidden state training mean, and hidden state training variance corresponding to each hidden layer into the following calculation formula to obtain the alignment loss:

[0029] Substituting the hidden state test mean, hidden state test variance, hidden state training mean, and hidden state training variance for each hidden layer into the following formula, we obtain the alignment loss:

[0030]

[0031] In the formula, L align Let L represent the alignment loss, and L represent the total number of hidden layers. This represents the test mean of the hidden states of the l-th hidden layer. This represents the test variance of the hidden state in the l-th hidden layer. This represents the training mean of the hidden states of the l-th hidden layer. Let represent the training variance of the hidden state of the l-th hidden layer.

[0032] Thirdly, embodiments of this application provide a multi-task prompting decision converter construction device, the multi-task prompting decision converter construction device including a processor, a memory, and a multi-task prompting decision converter construction program stored in the memory and executable by the processor, wherein when the multi-task prompting decision converter construction program is executed by the processor, it implements the steps of the multi-task prompting decision converter construction method as described above.

[0033] Fourthly, embodiments of this application provide a computer-readable storage medium storing a multi-task prompting decision converter construction program, wherein when the multi-task prompting decision converter construction program is executed by a processor, it implements the steps of the aforementioned multi-task prompting decision converter construction method.

[0034] The beneficial effects of the technical solutions provided in this application include:

[0035] During the adaptation phase, most decision layers in the MPDT are strategically frozen, while only a small number of decision layers and the triplet prompting module built based on reward, state, and action are activated. This leverages a large amount of prior knowledge embedded in the language model to adapt to specific dataset tasks, ensuring the model can generalize to downstream robot control tasks. The MPDT is trained based on offline training samples, and during training, multi-task prompting fine-tuning is performed on the MPDT based on triplet prompts to learn specific task prompts for parameter effectiveness for each training task, while simultaneously learning general features across all training tasks, thus obtaining cross-task prompts. Then, in the testing phase, the combination of cross-task prompts and unlabeled test samples is used as input to the trained MPDT, and alignment operations are performed on the cross-task prompts to obtain alignment loss. Finally, backpropagation is performed on the MPDT based on the alignment loss to update the cross-task prompts. That is, cross-task prompt fine-tuning is achieved through zero-shot alignment and applied to unseen tasks, thereby achieving algorithm generalization in zero-shot scenarios and effectively improving the generalization ability of the converter in zero-shot and multi-task scenarios. Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating an embodiment of the multi-task prompting decision transformer construction method of this application;

[0037] Figure 2 This is a schematic diagram illustrating the multi-task prompting during the training phase in the embodiments of this application;

[0038] Figure 3 This is a schematic diagram illustrating the alignment of the testing phase in the embodiments of this application;

[0039] Figure 4 This is a schematic diagram of the model construction process involved in the embodiments of this application;

[0040] Figure 5 This is a schematic diagram of the hardware structure of the multi-task prompting decision converter construction device involved in the embodiments of this application. Detailed Implementation

[0041] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0043] In a first aspect, embodiments of this application provide a method for constructing a multi-task prompting decision transformer.

[0044] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the multi-task prompting decision transformer construction method of this application. Figure 1 As shown, the method for constructing a multi-task prompting decision transformer includes:

[0045] Step S10: Construct a Multi-task Prompt Decision Transformer (MPDT), which includes a multi-layer decision-maker layer and a triplet prompting module based on reward, state, and action.

[0046] Exemplary, in this embodiment, an MPDT comprising multiple decision layers will be constructed, with the multiple decision layers forming the main body of the MPDT. Considering the differences in downstream tasks between reinforcement learning and natural language processing, the prompting component is designed as a triplet prompting module in the form of triplets, ensuring that the prompts can capture corresponding information during the learning process; that is, the MPDT also includes a triplet prompting module. The triplet form is... and These represent reward cues, status cues, and action cues, respectively, used to guide the model in generating relevant outputs.

[0047] Therefore, in the model initialization phase, this embodiment will initialize the decision layer and triplet prompting module as the base model of MPDT.

[0048] Furthermore, in one embodiment, before the step of training the MPDT based on preset offline training samples, the method further includes:

[0049] The multi-layer decision maker is initialized based on the pre-trained language model GPT2-SMALL to determine the parameter weights in the multi-layer decision maker.

[0050] As an example, in this embodiment, to fully utilize the few-sample generalization and extrapolation capabilities of the language model (by adding information such as the number of model layers), the main part of the MPDT architecture (i.e., the multi-layer decision maker layer) will be initialized using the pre-trained language model GPT2-SMALL. This updates and determines the parameter weights in the multi-layer decision maker layer, thereby effectively improving the model's generalization ability. GPT2-SMALL is a smaller version of the GPT series models, capable of generating high-quality, fluent, and coherent text, with faster inference speed and lower resource consumption.

[0051] Step S20: Randomly select at least one target decision layer from the multi-layer decision layer, control the target decision layer and the triplet prompting module to be in an active state, and train the GPT based on preset offline training samples. During the training process, fine-tune the GPT based on triplet prompts to generate specific task prompts and cross-task prompts.

[0052] In this exemplary embodiment, the primary objective of the model adaptation phase is to enable the model to learn to produce outputs for specific needs when prompted to participate. See also... Figure 2 As shown, in order to leverage the vast amount of prior knowledge embedded in the language model, strategic freezing ( Figure 2 In the multi-layer decision-maker system (using snowflakes to represent freezing), most decision-maker layers are only activated ( Figure 2 The model activates a small subset of decision layers (represented by flames) and triplet hint modules to adapt to specific dataset tasks. Specifically, freezing most decision layers ensures the model possesses sufficient out-of-domain knowledge, while activating a small subset ensures the model can generalize to downstream robot control tasks. It should be noted that the number and specific decision layers to activate can be determined based on actual needs or randomly, and are not limited here.

[0053] Understandably, reinforcement learning, as an important branch of artificial intelligence and machine learning, has been widely applied in various fields such as games, robot navigation, and autonomous driving. However, traditional online reinforcement learning methods rely on real-time interaction with the environment to acquire data and learn strategies. In many practical applications, real-time interaction is either too costly or poses security risks, making large-scale deployment of online learning difficult. In this embodiment, offline reinforcement learning will be used to train the MPDT, that is, to control the MPDT to be trained on the entire offline dataset. Specifically, based on pre-defined offline training samples with existing labels (… Figure 2 The sum of reward states in The environment state o and action state a) train the MPDT; it should be understood that, in this embodiment, the primary goal of the adaptation phase is to promote the model’s ability to adjust its output according to a specific dataset and sample, so no distinction is made between tasks in different scenarios, and the risk of convergence challenges due to gradient conflicts of different tasks will be resolved in subsequent steps. This strategy lays a solid foundation for subsequent stages.

[0054] During the training phase, the same loss function as that used for training large models can be employed.

[0055]

[0056] In the formula, t0 represents the starting point of the observation sequence, D represents the length of the observation sequence, and a t a′ represents the actual action taken at time t. t This represents the actions predicted by the model at the same time point. (Through...) Perform backpropagation to update the parameter weights of the multi-task decision maker layer.

[0057] During the multi-task prompting fine-tuning phase, the multi-layer decision-maker is not trained; only the triplet prompting module is trained, i.e., only multi-task prompting is trained. Specifically, for a given set of robot training tasks S = (S1, S2, ..., S... n The goal of multi-task cue fine-tuning is to learn specific task-specific cues P for the effectiveness of parameters for each training task based on training samples. z Simultaneously, during training, it learns general features across all training tasks, obtaining cross-task cues P. c This is used for prompting subsequent test tasks. It is understandable that much of the work done in previous multi-task prompt fine-tuning phases focused on learning a different prompt for each task, without considering the correlation between tasks in a multi-task scenario. In this embodiment, however, cross-task prompts P... c =(P rc ,P oc ,P ac The aim is to extract shared knowledge from training task S, while specific task cues P z =(P rz ,P oz ,P az This allows each task to retain its unique knowledge.

[0058] Furthermore, in one embodiment, the specific task prompt is composed of a first low-rank vector and a second low-rank vector, wherein the first low-rank vector is determined based on the prompt length and a low-rank parameter, and the second low-rank vector is determined based on the prompt dimension and the low-rank parameter.

[0059] As an example, in this embodiment, to reduce computational complexity, the specific task cue is further decomposed into two low-rank vectors: the first low-rank vector v z ∈l*r and the second low-rank vector u z ∈r*s, and then obtain the task-specific cue P through low-rank methods and vector multiplication. zHere, l represents the cue length, s represents the cue dimension, and the hyperparameter r is a manually specified low-rank parameter whose specific value can be determined according to actual needs and is not limited here. It should be understood that the introduction of the hyperparameter r is crucial for designing cueing for all tasks in the dataset, significantly reducing computational load while maintaining model superiority. Based on this, see [reference needed]. Figure 2 As shown, the parameterization of the three task cues for the k-th training task can be expressed as follows:

[0060]

[0061]

[0062]

[0063] Step S30: After the MPDT training is completed, the combination of cross-task prompts and unlabeled test samples is used as the input of the MPDT for testing, and the hidden state test mean and hidden state test variance corresponding to the test samples are calculated.

[0064] The step of calculating the hidden state test mean and hidden state test variance corresponding to the test sample includes: for each hidden layer in the MPDT, calculating the hidden state test mean based on the hidden state of each test sample in the hidden layer, and calculating the hidden state test variance based on the hidden state test mean.

[0065] In this exemplary embodiment, the MPDT is tested after training is complete. It is worth noting that during testing, this embodiment uses unlabeled test samples to perform the MPDT test. Since the test samples are not labeled, it can be designed as a zero-shot learning task. During the testing phase, considering its zero-shot nature, the distribution of real-time test samples is aligned with the distribution of training samples within the pre-trained model. This addresses distribution biases that occur in suboptimal test sets, enabling fine-tuning of prompts in zero-shot scenarios. This is crucial for handling distribution changes that occur in offline reinforcement learning environments. It should be noted that the test samples in this embodiment are selected from an online environment, and zero-shot fine-tuning of cross-task prompts is performed based on these test samples in offline scenarios.

[0066] Specifically, for each test task, N robot behavior trajectory samples are randomly selected as test samples. Cross-task cues obtained during training are combined with these test samples, and the combination result is used as input to the MPDT. Then, for each hidden layer in the MPDT, the state mean and state variance of the test samples are calculated. Specifically, assuming the distribution of the test samples is represented by T, the mean (i.e., hidden state test mean) and variance (i.e., hidden state test variance) of each hidden state layer can be calculated using the following formulas:

[0067]

[0068]

[0069] In the formula, H represents the mean of the hidden states at layer l. l,i This represents the hidden state of the i-th test sample in layer l. This represents the variance of the hidden state at layer l.

[0070] Step S40: Perform alignment calculation based on the hidden state test mean and hidden state test variance, as well as the hidden state training mean and hidden state training variance corresponding to the offline training samples, to obtain the alignment loss.

[0071] The step of calculating the alignment loss based on the hidden state test mean and hidden state test variance, as well as the hidden state training mean and hidden state training variance corresponding to the offline training samples, includes: substituting the hidden state test mean, hidden state test variance, hidden state training mean, and hidden state training variance corresponding to each hidden layer into the following calculation formula to obtain the alignment loss:

[0072]

[0073] In the formula, L align Let L represent the alignment loss, and L represent the total number of hidden layers. This represents the test mean of the hidden states of the l-th hidden layer. This represents the test variance of the hidden state in the l-th hidden layer. This represents the training mean of the hidden states of the l-th hidden layer. Let represent the training variance of the hidden state of the l-th hidden layer.

[0074] Exemplary, see Figure 3As shown, the alignment loss between the state mean and state variance of the training and test samples will be calculated in each hidden layer of the MPDT to update the cross-task cue for the test samples. Importantly, since the training samples and labels are available, the training mean and hidden state variance of the training samples can be pre-calculated during offline training to reduce parallel computation costs. It should be noted that the calculation method and principle for the training mean and hidden state variance of the training samples are the same as those for the test mean and hidden state variance of the test samples; for the sake of simplicity, they will not be repeated here.

[0075] Assuming the distribution of the training samples is represented by D, then the training mean of the hidden states of the training samples is... And the training variance of the hidden state is Therefore, the mean value of the hidden state corresponding to each hidden layer is tested. Hidden State Test Variance Hidden state training mean and hidden state training variance Substituting into the following formula, the alignment loss L can be obtained. align :

[0076]

[0077] In the formula, L represents the total number of hidden layers.

[0078] Step S50: Backpropagate the MPDT based on the alignment loss to update cross-task cues and generate a target multi-task cue decision transformer.

[0079] As an example, in this embodiment, the loss L will be optimized. align To update cross-task hints, i.e. based on alignment loss L align Backpropagation of the MPDT is performed to update cross-task prompts, thereby generating a target multi-task prompt decision transformer. This achieves algorithm generalization in zero-shot scenarios, allowing the generated target multi-task prompt decision transformer to be used in specific unseen test tasks in zero-shot scenarios. Therefore, this embodiment provides a multi-task prompt decision transformer for zero-shot generalization scenarios. This transformer achieves high parameter utilization and excellent cross-task generalization performance by integrating pre-trained language model initialization techniques with multi-task prompt adjustment strategies.

[0080] In summary, this embodiment innovatively introduces prompt-based offline reinforcement learning into the field of robot control, proposing a multi-task prompt-decision transformer based on a transformer architecture. This transformer can adapt to knowledge learning in multi-task scenarios and generalize to zero-shot scenarios, solving the problem of zero-shot precise robot motion control in multi-task scenarios. Furthermore, this embodiment utilizes a pre-trained language model to design a three-stage method to enhance the model's prior knowledge, enabling faster algorithm convergence. Simultaneously, in the multi-task prompt fine-tuning stage, multi-task prompts are decomposed into cross-task prompts and specific task prompts, and alignment operations are performed on cross-task prompts during the testing phase, achieving zero-shot generalization. Therefore, this embodiment, based on the combination of test-time adaptation and prompt adjustment, jointly improves the model's versatility and adaptability in zero-shot scenarios. It provides a powerful framework for future exploration and optimization of multi-task learning and generalization capabilities, helping to better focus on the model's generalization ability and efficiency, especially in scenarios with limited data or more complex tasks.

[0081] The following examples are combined Figure 4 The construction method and principle of the target multi-task prompting decision transformer are explained.

[0082] First, in the initialization phase, the offline dataset for the robot simulation environment is defined as follows: A multi-task cueing decision transformer structure is defined, consisting of 12 decision layers, each with 12 attention heads. Considering the differences between reinforcement learning and natural language processing in downstream tasks, random initialization prompts Among them, P rk Corresponding reward prompt, P ok Corresponding status message, P ak The triplet format for corresponding action cues ensures that the cues capture the relevant information during the learning process; simultaneously, to fully utilize the generalization ability of the pre-trained language model, GPT2-SMALL is used for initialization.

[0083] During the adaptation phase, the control model is trained on the entire offline dataset and then frozen. Layers 0, 1, 3, 4, 5, 6, 8, 9, 10, and 11, and activated. The 2nd and 7th layers, as well as the triplet hint module. Understandably, this is frozen. The purpose of most hidden layers is to ensure that the model has sufficient out-of-domain knowledge, while activation... The purpose of the small number of hidden layers is to ensure that the model can generalize to downstream robot control tasks. It should be noted that at this stage, no distinction is made between tasks in different scenarios. The primary goal of this process is to promote the model's ability to adjust its output according to specific datasets and example patterns. The risk of convergence challenges due to gradient conflicts between different tasks will be resolved in subsequent steps.

[0084] During the multi-task prompting fine-tuning phase, for a given set of robot training tasks S = (S1, S2, ..., S...), n The goal of multi-task cue fine-tuning is to learn task-specific cues P for parameter effectiveness for each training task based on an offline dataset. z Simultaneously, during training, it learns general features across all training tasks, obtaining cross-task cues P. c This is used as a prompt for subsequent test tasks. Specifically, cross-task prompts P c =(P rc ,P oc ,P ac The aim is to extract shared knowledge from S, while specific task hints are given to P. z =(P rz ,P oz ,P az This allows each task to retain its unique knowledge.

[0085] During the testing phase, the model is tested using a test dataset without test labels, thus aligning the distribution of real-time test data with the distribution of the training dataset within the pre-trained model. Specifically, for each test task, N robot behavior trajectory samples are randomly selected as test samples, and the cross-task cues obtained during training are combined with the test samples as model input. Furthermore, the alignment loss between the mean and variance of the training and test samples is calculated in each hidden layer of the model. Finally, the cross-task cues of the test samples are updated using the alignment loss to output the final target multi-task cue decision transformer. This target multi-task cue decision transformer can then be applied to real-world simulation environments for robot motion control, etc.

[0086] Understandably, in this embodiment, multi-task offline reinforcement learning allows the model to process multiple tasks simultaneously and facilitates the transfer of knowledge learned in a single task to other tasks, thereby achieving cross-task transfer learning. Existing algorithms mainly achieve downstream task generalization through pre-training and fine-tuning on the overall dataset, without considering conflicts and commonalities between tasks. However, this embodiment effectively integrates pre-trained language model parameters and the offline reinforcement learning framework by implementing a three-stage paradigm learning approach during training to provide cross-task and task-specific multi-task cues. Furthermore, a cue alignment strategy is employed to achieve zero-shot generalization of robot motion trajectory labels during the testing phase, thereby improving parameter efficiency and enhancing cross-task generalization capabilities, enabling the model to effectively handle unknown tasks.

[0087] The advantages of the multi-task prompting decision converter provided in this embodiment will be further illustrated below with comparative experiments.

[0088] First, this embodiment will conduct experiments on multi-task behavior cloning, multi-task decision transformer, cue decision transformer, and the multi-task cue decision transformer of this embodiment based on the Cheetah-Direction Dataset, Cheetah-Speed ​​Dataset, Metaworld 10 Dataset, and Metaworld 45 Dataset, respectively.

[0089] It should be noted that the Cheetah-Direction dataset contains two tasks with target directions of forward and backward. The cheetah agent receives a reward when it moves at high speed in the target direction. In this embodiment, the training and test sets are equal and include both tasks.

[0090] Cheetah-Speed ​​Dataset: The Cheetah-Speed ​​dataset contains 40 tasks with different target velocities, uniformly sampled from the interval [0,3]. The agent is penalized for L2 errors in the target velocities. This embodiment uses 5 tasks to construct the test set and the remaining 35 tasks for training.

[0091] Metaworld 10 Dataset: In the Metaworld 10 dataset, the task is to control the end effector of a Sawyer robot to reach a target position in 3D space; the agent directly controls the XYZ position of the end effector, and each task has a different target position. This embodiment will be trained on 10 tasks and tested on 5 unseen tasks.

[0092] Metaworld 45 Dataset: In the Metaworld 45 dataset, the task is to control the end effector of a Sawyer robot to reach a target position in 3D space, and each task has a different target position. This example will be trained on 45 tasks and tested on 5 unseen tasks.

[0093] The experimental data for the reward parameter, as shown in Table 1, were obtained after the above experiments.

[0094] Table 1 Experimental Data

[0095]

[0096] The experimental data above shows that the multi-task cueing decision transformer provided in this embodiment outputs better reward values ​​than other transformers on various datasets; thus, the generalization ability of the multi-task cueing decision transformer provided in this embodiment is better than that of other transformers.

[0097] Secondly, embodiments of this application also provide a multi-task prompting decision converter construction apparatus.

[0098] In one embodiment, the multi-task prompting decision transformer construction apparatus includes: a construction module, a training module, and a testing module;

[0099] The building module is used to build a multi-task cue decision transformer (MPDT), which includes a multi-layer decision transformer layer and a triple cue module based on reward, state, and action.

[0100] The training module is used to randomly select at least one target decision layer from the multi-layer decision layer, control the target decision layer and the triplet prompting module to be in an active state, and train the MPDT based on preset offline training samples. During the training process, the MPDT is fine-tuned for multi-task prompting based on triplet prompts to generate specific task prompts and cross-task prompts.

[0101] The testing module is used to test the combination of the cross-task prompts and unlabeled test samples as input to the MPDT after the MPDT training is completed, and to calculate the hidden state test mean and hidden state test variance corresponding to the test samples; and to perform alignment calculation based on the hidden state test mean and hidden state test variance and the hidden state training mean and hidden state training variance corresponding to the offline training samples to obtain the alignment loss.

[0102] The building module is also used to backpropagate the MPDT based on the alignment loss to achieve cross-task cue updates and generate a target multi-task cue decision transformer.

[0103] Furthermore, in one embodiment, the building module is also used to: initialize the multi-layer decision maker based on the pre-trained language model GPT2-SMALL to determine the parameter weights in the GPT.

[0104] Furthermore, in one embodiment, the specific task prompt is composed of a first low-rank vector and a second low-rank vector, wherein the first low-rank vector is determined based on the prompt length and a low-rank parameter, and the second low-rank vector is determined based on the prompt dimension and the low-rank parameter.

[0105] Furthermore, in one embodiment, the testing module is specifically used for:

[0106] For each hidden layer in the MPDT, the hidden state test mean is calculated based on the hidden state of each test sample in the hidden layer, and the hidden state test variance is calculated based on the hidden state test mean.

[0107] Furthermore, in one embodiment, the testing module is specifically used for:

[0108] Substituting the hidden state test mean, hidden state test variance, hidden state training mean, and hidden state training variance for each hidden layer into the following formula, we obtain the alignment loss:

[0109]

[0110] In the formula, L align Let L represent the alignment loss, and L represent the total number of hidden layers. This represents the test mean of the hidden states of the l-th hidden layer. This represents the test variance of the hidden state in the l-th hidden layer. This represents the training mean of the hidden states of the l-th hidden layer. Let represent the training variance of the hidden state of the l-th hidden layer.

[0111] The functions of each module in the above-mentioned multi-task prompting decision converter construction device correspond to the steps in the above-mentioned multi-task prompting decision converter construction method embodiment, and their functions and implementation processes will not be described in detail here.

[0112] Thirdly, embodiments of this application provide a multi-task prompting decision converter construction device, which can be a device with data processing capabilities such as a personal computer (PC), a laptop computer, or a server.

[0113] Reference Figure 5 , Figure 5 This is a schematic diagram of the hardware structure of the multi-task prompting decision converter construction device involved in the embodiments of this application. In the embodiments of this application, the multi-task prompting decision converter construction device may include a processor, a memory, a communication interface, and a communication bus.

[0114] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.

[0115] The communication interface includes input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting devices within the multi-task prompting decision converter building device, as well as interfaces used for interconnecting the multi-task prompting decision converter building device with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.

[0116] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0117] The processor can be a general-purpose processor, which can call the multi-task prompting decision transformer construction program stored in memory and execute the multi-task prompting decision transformer construction method provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the multi-task prompting decision transformer construction program is called can be referred to in the various embodiments of the multi-task prompting decision transformer construction method of this application, and will not be repeated here.

[0118] Those skilled in the art will understand that Figure 5 The hardware structure shown does not constitute a limitation of this application and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0119] Fourthly, embodiments of this application also provide a computer-readable storage medium.

[0120] This application stores a multi-task prompting decision transformer construction program on a readable storage medium, wherein when the multi-task prompting decision transformer construction program is executed by a processor, it implements the steps of the multi-task prompting decision transformer construction method as described above.

[0121] The method implemented when the multi-task prompting decision transformer construction program is executed can be referred to in various embodiments of the multi-task prompting decision transformer construction method of this application, and will not be repeated here.

[0122] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0123] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0124] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0125] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0126] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0127] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0128] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for constructing a multi-task prompting decision transformer, characterized in that, Includes the following steps: A multi-task cueing decision transformer (MPDT) is constructed, which includes a multi-layer decision transformer layer and a triple cueing module based on reward, state, and action. At least one target decision layer is randomly selected from the multi-layer decision layer, and the target decision layer and the triplet prompting module are kept in an active state. The MPDT is trained based on a preset offline training sample. During the training process, the MPDT is fine-tuned based on the triplet prompts to generate specific task prompts and cross-task prompts. After the MPDT training is completed, the combination of the cross-task prompts and unlabeled test samples is used as the input of the MPDT for testing, and the hidden state test mean and hidden state test variance corresponding to the test samples are calculated. Alignment loss is obtained by performing alignment calculation based on the hidden state test mean and hidden state test variance, as well as the hidden state training mean and hidden state training variance corresponding to the offline training samples. Backpropagation of the MPDT is performed based on the alignment loss to achieve cross-task cue updates and generate a target multi-task cue decision transformer. The step of calculating the alignment loss based on the hidden state test mean and hidden state test variance, and the hidden state training mean and hidden state training variance corresponding to the offline training samples, includes: Substituting the hidden state test mean, hidden state test variance, hidden state training mean, and hidden state training variance for each hidden layer into the following formula, we obtain the alignment loss: In the formula, Indicates alignment loss. This indicates the total number of hidden layers. Indicates the first The hidden state test mean of each hidden layer Indicates the first The test variance of the hidden state of each hidden layer Indicates the first The training mean of the hidden states of each hidden layer, Indicates the first The training variance of the hidden states of each hidden layer.

2. The method for constructing a multi-task prompting decision transformer as described in claim 1, characterized in that, Before the step of training the MPDT based on preset offline training samples, the method further includes: The multi-layer decision maker is initialized based on the pre-trained language model GPT2-SMALL to determine the parameter weights in the multi-layer decision maker.

3. The method for constructing a multi-task cueing decision transformer as described in claim 1, characterized in that: The specific task prompt consists of a first low-rank vector and a second low-rank vector. The first low-rank vector is determined based on the prompt length and the low-rank parameter, and the second low-rank vector is determined based on the prompt dimension and the low-rank parameter.

4. The method for constructing a multi-task prompting decision transformer as described in claim 1, characterized in that, The calculation of the hidden state test mean and hidden state test variance corresponding to the test sample includes: For each hidden layer in the MPDT, the hidden state test mean is calculated based on the hidden state of each test sample in the hidden layer, and the hidden state test variance is calculated based on the hidden state test mean.

5. A device for constructing a multi-task prompting decision transformer, characterized in that, The multi-task prompting decision converter construction device includes: a construction module, a training module, and a testing module; The building module is used to build a task prompting decision transformer (MPDT), which includes a multi-layer decision transformer layer and a triple prompting module based on reward, state, and action. The training module is used to randomly select at least one target decision layer from the multi-layer decision layer, control the target decision layer and the triplet prompting module to be in an active state, and train the MPDT based on preset offline training samples. During the training process, the MPDT is fine-tuned for multi-task prompting based on triplet prompts to generate specific task prompts and cross-task prompts. The testing module is used to test the combination of the cross-task prompts and unlabeled test samples as input to the MPDT after the MPDT training is completed, and to calculate the hidden state test mean and hidden state test variance corresponding to the test samples; and to perform alignment calculation based on the hidden state test mean and hidden state test variance and the hidden state training mean and hidden state training variance corresponding to the offline training samples to obtain the alignment loss. The building module is also used to backpropagate the MPDT based on the alignment loss to achieve cross-task cue updates and generate a target multi-task cue decision transformer. Specifically, the testing module is used for: Substituting the hidden state test mean, hidden state test variance, hidden state training mean, and hidden state training variance for each hidden layer into the following formula, we obtain the alignment loss: In the formula, in the formula, Indicates alignment loss. This indicates the total number of hidden layers. Indicates the first The hidden state test mean of each hidden layer Indicates the first The test variance of the hidden state of each hidden layer Indicates the first The training mean of the hidden states of each hidden layer, Indicates the first The training variance of the hidden states of each hidden layer.

6. The multi-task prompting decision transformer construction apparatus as described in claim 5, characterized in that, The building module is also used for: The multi-layer decision maker is initialized based on the pre-trained language model GPT2-SMALL to determine the parameter weights of each layer.

7. The multi-task prompting decision transformer construction apparatus as described in claim 5, characterized in that: The specific task prompt consists of a first low-rank vector and a second low-rank vector. The first low-rank vector is determined based on the prompt length and the low-rank parameter, and the second low-rank vector is determined based on the prompt dimension and the low-rank parameter.

8. A device for constructing a multi-task prompting decision transformer, characterized in that, The multi-task prompting decision converter construction device includes a processor, a memory, and a multi-task prompting decision converter construction program stored in the memory and executable by the processor, wherein when the multi-task prompting decision converter construction program is executed by the processor, it implements the steps of the multi-task prompting decision converter construction method as described in any one of claims 1 to 4.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a multi-task prompting decision transformer construction program, wherein when the multi-task prompting decision transformer construction program is executed by a processor, it implements the steps of the multi-task prompting decision transformer construction method as described in any one of claims 1 to 4.

Images