Action generation method and related apparatus

Abstract

Provided in the embodiments of the present application are an action generation method and a related apparatus. The method comprises: generating a first sequence on the basis of input information, the input information comprising a task instruction and an initial image; performing a preset processing on the first sequence to generate a second sequence, the second sequence comprising a plurality of token sets classified according to modalities, and the plurality of token sets being in one-to-one correspondence with a plurality of types of information in the input information; on the basis of an attention mechanism, processing the second sequence to generate a third sequence, the third sequence comprising bidirectional association information of tokens in each of the plurality of token sets, and comprising causal association information of tokens in different token sets among the plurality of token sets; and generating a target action on the basis of the third sequence, the target action being an action for executing a target task. The embodiments of the present application can improve the accuracy of generated actions.

Description

Action generation method and related device

[0001] The present application claims priority from the Chinese patent application No. 202410869406.7 filed on June 29, 2024, and entitled "Action generation method and related device", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of embodied intelligence, and in particular, to an action generation method and related device. BACKGROUND

[0003] With the rapid development of deep learning and other technologies, the field of embodied artificial intelligence (embodied AI) pays more and more attention to the ability of embodied entities (such as robots, mechanical arms, etc.) to make autonomous decisions and perform physical interaction tasks. Currently, embodied entities can understand language instructions and perceive the environment with the help of vision-language-action models (VLAs), and perform corresponding actions to complete target tasks. Existing VLAs are usually action generation models based on the Transformer architecture, therefore, the accuracy of the actions generated by the model directly affects the accuracy of the embodied entities when completing the specified tasks. SUMMARY

[0004] Embodiments of the present application provide an action generation method and related device, which can be applied to the field of embodied intelligence to generate actions required to perform target tasks, and can improve the accuracy of the generated actions.

[0005] In a first aspect, embodiments of the present application provide an action generation method, which comprises:

[0006] generating a first sequence according to input information, wherein the input information comprises multiple types of information, the multiple types of information comprising a task instruction and an initial image, the task instruction being used to indicate a target task, and the initial image being used to reflect environmental information corresponding to the target task;

[0007] performing a preset processing on the first sequence to generate a second sequence, wherein the second sequence comprises multiple token sets classified according to modalities, and the multiple token sets correspond one-to-one to the multiple types of information in the input information;

[0008] performing processing on the second sequence based on the attention mechanism to generate a third sequence, wherein the third sequence comprises bidirectional association information of tokens in each token set and causal association information between tokens in different token sets;

[0009] generating the target action based on the third sequence, wherein the target action belongs to an action of performing the target task.

[0010] In the above method, considering that the various information in the input information (such as the task instruction and the initial image) belongs to information of different modalities in the embodied intelligence scene, and the information of different modalities has strong causal association, and the information of a single modality itself has strong bidirectional association, therefore, the above method first classifies the input information by modality to obtain a plurality of token sets, each token set is used to reflect the information of a modality, and then processes the plurality of token sets through the attention mechanism to extract the causal relationship between the information of different modalities and the bidirectional association of the content of the information of a single modality, and then generates the target action based on the third sequence containing these relationships, thereby avoiding the jump and obvious distortion of the action, and further improving the accuracy of the generated action.

[0011] In an optional implementation, the input information further comprises the target action and an incremental image, and after generating the target action based on the third sequence, the method further comprises:

[0012] If the target task is not completed after performing the target action, the target action and the incremental image are added to the input information, and the incremental image is used to reflect the environmental information after performing the target action;

[0013] Return to the operation of generating the first sequence based on the input information.

[0014] For a target task that needs to perform multiple actions to complete, the above method adds the action and the environmental information after performing the action to the existing input information after generating an action each time, and then performs the operation of generating a new action based on the new input information, and uses this iterative way until the target task is completed. It can be seen that in the above method, the generation of each new action is based on the previously generated all actions and the environmental information after performing all actions as input information, so that the above method can guarantee the association and continuity between the actions, thereby guaranteeing the accuracy of each action.

[0015] In yet another optional implementation, the target action comprises a plurality of action tokens, and the plurality of action tokens correspond to actions of the target action in a plurality of action dimensions, respectively.

[0016] If the action of the target action in one action dimension of the plurality of action dimensions is a preset action token, the target task is not completed after the target action is performed.

[0017] In yet another optional implementation, the plurality of token sets are arranged according to an interaction relationship between the plurality of information in the input information, and the interaction relationship includes a relationship between the task instruction and the initial image generation action.

[0018] In the above method, considering that in the embodied intelligence scene, there is a certain interaction relationship (which can also be understood as a causal relationship) between the plurality of information in the input information, such as collecting initial image information after obtaining a task instruction, such as obtaining an action according to a task instruction and an initial image, such as collecting environment information after performing an action, therefore, the above method sorts the plurality of token sets corresponding to the plurality of information in the second sequence according to the interaction relationship, facilitating subsequent accurate extraction of the causal relationship between the plurality of token sets by using the attention mechanism, thereby improving the accuracy of the generated action.

[0019] In yet another optional implementation, the first sequence is preprocessed to generate the second sequence, including:

[0020] The first sequence is processed based on a Markov decision process to generate the second sequence.

[0021] The above method classifies the plurality of tokens in the first sequence into modalities and arranges them by means of the mathematical framework of the Markov decision process, facilitating generation of the third sequence.

[0022] In yet another optional implementation, the second sequence includes a text token set, m image token sets, and m-1 action token sets, the text token set corresponds to a task instruction in the input information, the i-th image token set in the m image token sets corresponds to the i-th image in the input information, the j-th action token set in the m-1 action token sets corresponds to the j-th action in the input information, the text token set is a target constraint in the Markov decision process, the i-th image token set is a state at the i-th time in the Markov decision process, and the j-th action token set is an action at the j-th time in the Markov decision process.

[0023] In yet another optional implementation, the second sequence is processed based on an attention mechanism to generate the third sequence, including:

[0024] The second sequence is processed by the causal attention mechanism and the bidirectional attention mechanism to generate the fifth sequence.

[0025] The method adds a set of action query parameters after each token set corresponding to an image in the input information in the fifth sequence to generate a third sequence, wherein the set of action query parameters is used to query the relationship between the plurality of token sets and the action.

[0026] The method associates the third sequence with the action through embedding of the set of action query parameters, and facilitates subsequent generation of the target action according to the third sequence.

[0027] In another optional implementation, the method of generating the target action according to the third sequence comprises:

[0028] The transformer network is used to perform autoregressive processing on the third sequence to generate the target action.

[0029] In another optional implementation, the target action includes n action tokens, the set of action query parameters includes n action query parameters, and the kth action query parameter in the last set of action query parameters in the third sequence is used to extract relevant information of the kth action token in the n action tokens in the third sequence, wherein n is a preset action dimension number, k is 1, 2, …, n, and n is a positive integer.

[0030] The method extracts information of multiple modalities most relevant to sub-actions of the target action in multiple action dimensions in the input information through multiple action query parameters, which not only enhances the flow of information conducive to generation of the target action, further improves the accuracy of the generated action, but also realizes display alignment between multiple modalities of information in the input information and sub-actions in different dimensions, and synchronously generates multiple action tokens in a target action, thereby improving the efficiency of action generation.

[0031] In another optional implementation, the method is implemented based on an action generation model, the action generation model is used to perform the operation of generating the target action according to the input information, and the action generation model is obtained from a plurality of training samples, wherein the plurality of training samples include positive samples and negative samples.

[0032] In another optional implementation, the action generation model is used to:

[0033] generate a first sequence according to the input information;

[0034] perform a preset processing on the first sequence to generate a second sequence;

[0035] perform processing on the second sequence based on an attention mechanism to generate a third sequence;

[0036] generate the target action according to the third sequence.

[0037] In another alternative implementation, the positive sample includes a first positive sample and a second positive sample. The first positive sample includes a task instruction, an initial image, and a reference action. The second positive sample includes a task instruction, an initial image, and an action obtained by noise processing of the reference action. The reference action is a standard action for performing the target task.

[0038] The above method improves the model's ability to recognize correct actions by introducing positive samples, which is beneficial for improving the accuracy of generated actions based on multiple training positive samples.

[0039] In another alternative implementation, the negative samples include one or more of a first negative sample, a second negative sample, and a third negative sample, wherein:

[0040] The first negative sample includes a interference instruction, an initial image, and a reference action. The interference instruction is used to instruct the execution of a task different from the target task.

[0041] The second negative sample includes task instructions, interference images, and reference actions. The interference images are used to reflect the environmental information corresponding to performing a task that is different from the target task.

[0042] The third negative sample includes the task instruction, the initial image, and the interfering action, which is the standard action corresponding to performing a task different from the target task.

[0043] The above method processes text information, image information, and action information separately according to the modality of the information to construct multiple negative samples. By introducing multiple negative samples, the model's ability to identify erroneous actions is improved. This is beneficial for improving the action generation model's ability to align information of different modalities through comparative learning, thereby improving the accuracy of the generated actions.

[0044] Secondly, embodiments of this application provide a model training method, the action generation method comprising:

[0045] Multiple first training sequences are generated based on multiple training samples. The multiple training samples include first positive samples, which include task instructions, initial images, and reference actions. The task instructions are used to instruct the execution of the target task, the initial images are used to reflect the environmental information corresponding to the execution of the target task, and the reference actions are the standard actions for executing the target task.

[0046] The first training sequence is pre-processed to generate the second training sequence, wherein the second training sequence includes multiple token sets classified according to modality, and the multiple token sets correspond one-to-one with various information in the training samples;

[0047] The second training sequence is processed based on an attention mechanism to generate a third training sequence, where the third training sequence includes bidirectional association information of tokens in each token set in the plurality of token sets, and includes causal association information of tokens between different token sets in the plurality of modal token sets;

[0048] The predicted action is generated according to the third training sequence;

[0049] Parameters in the action generation model are optimized according to the reference action and the predicted action, where the action generation model is used to generate an action for performing a target task according to input information, and the input information includes a task instruction and an initial image.

[0050] In the above method, considering that the various information (such as the task instruction and the initial image) in the training sample belongs to information of different modalities in the embodied intelligence scene, and the information of different modalities has strong causal association, and the content of the information of a single modality itself has strong bidirectional association, the above method first classifies the training sample by modality to obtain a plurality of token sets, each token set is used to reflect information of a modality, and then processes the plurality of token sets through an attention mechanism to extract the causal relationship between the information of different modalities and the bidirectional association relationship of the content of the information of a single modality, and then generates a training action based on the third training sequence containing these relationships, thereby avoiding the action from jumping and being obviously distorted, and further improving the accuracy of the generated predicted action. Therefore, the above method can converge the action generation model faster and improve the model training efficiency.

[0051] In an optional implementation, the plurality of training samples includes a second positive sample, and the second positive sample includes a task instruction, an initial image, and an action obtained by performing noise processing on the reference action.

[0052] The above method improves the recognition degree of the model for the correct action by introducing a plurality of positive samples, which is beneficial to improve the accuracy of the generated action.

[0053] In another optional implementation, the plurality of training samples further includes one or more of a first negative sample, a second negative sample, and a third negative sample, where:

[0054] The first negative sample includes an interference instruction, an initial image, and a reference action, and the interference instruction is used to instruct to perform a task different from the target task;

[0055] The second negative sample includes a task instruction, an interference image, and a reference action, and the interference image is used to reflect environmental information corresponding to performing a task different from the target task;

[0056] The third negative sample includes a task instruction, an initial image, and an interference action, and the interference action is a standard action corresponding to a task different from the target task.

[0057] The method processes text information, image information, and action information according to modalities of the information, thereby introducing multiple negative samples. By introducing multiple negative samples, the recognition of the model for incorrect actions is improved, which is conducive to improving the alignment capability of the action generation model for different modal information through a contrast learning manner, thereby improving the accuracy of the generated actions.

[0058] In yet another optional implementation, the first positive sample further includes m-1 incremental images, the reference action includes m reference actions, and the predicted action includes m predicted actions. Each of the m-1 incremental images is used to reflect environmental information after one of the m reference actions is performed, and the m predicted actions correspond to the m reference actions one by one.

[0059] In yet another optional implementation, if the m reference actions are performed in sequence, the target task is completed.

[0060] In yet another optional implementation, the first training sequence is preprocessed to generate a second training sequence, including:

[0061] The first training sequence is processed based on a Markov decision process to generate a second training sequence.

[0062] The method classifies multiple tokens in the first training sequence into modalities and arranges the tokens by means of the Markov decision process, which facilitates generation of a third training sequence.

[0063] In yet another optional implementation, the second training sequence includes a text token set, m image token sets, and m action token sets. The text token set corresponds to the task instruction in the first positive sample, the i-th image token set in the m image token sets corresponds to the i-th image in the first positive sample, the i-th action token set in the m action token sets corresponds to the i-th action in the first positive sample, the text token set is a target constraint in the Markov decision process, the i-th image token set is a state at the i-th time in the Markov decision process, and the i-th action token set is an action at the i-th time in the Markov decision process.

[0064] In yet another optional implementation, the second training sequence is processed based on an attention mechanism to generate a third training sequence, including:

[0065] The second training sequence is processed through a causal attention mechanism and a bidirectional attention mechanism to generate a fifth training sequence.

[0066] A set of action query parameters is added after each token set corresponding to the image in the first positive sample in the fifth training sequence to generate a third training sequence, wherein the set of action query parameters is used to query the relationship between the plurality of token sets and the action.

[0067] The above method associates the third training sequence with the action through embedding of the set of action query parameters, facilitating subsequent generation of a predicted action based on the third training sequence.

[0068] In yet another optional implementation, generating the predicted action based on the third training sequence comprises:

[0069] Performing autoregressive processing on the third training sequence based on a transformer network to generate the predicted action.

[0070] In yet another optional implementation, the predicted action includes m predicted actions, each of the m predicted actions includes n action tokens, the n action tokens correspond to actions of the predicted action in a plurality of action dimensions respectively, the set of action query parameters includes n action query parameters, and the kth action query parameter in the ith set of action query parameters in the third training sequence is used to extract relevant information of the kth action token of the n action tokens of the ith sample action in the m sample actions in the third sequence, n is a preset number of action dimensions, i takes 1, 2, …, m, k takes 1, 2, …, n, and m and n are positive integers.

[0071] The above method extracts information of multiple modalities most relevant to sub-actions of the predicted action in a plurality of action dimensions in the training sample through a plurality of action query parameters, which not only enhances the flow of information conducive to the generation of the predicted action, further improves the accuracy of the generated action, realizes the rapid convergence of the model, but also realizes the explicit alignment between the information of multiple modalities in the training sample and the sub-actions of different dimensions, and synchronously generates a plurality of action tokens in a predicted action, thereby improving the efficiency of action generation and further improving the efficiency of model training.

[0072] In a third aspect, the embodiments of the present application provide an action generation apparatus, which comprises a serialization module, a preprocessing module, a data modeling module, and an action generation module, wherein:

[0073] The serialization module is configured to generate a first sequence based on input information, wherein the input information comprises a task instruction and an initial image, the task instruction is used to instruct to perform a target task, and the initial image is used to reflect environmental information corresponding to the performance of the target task.

[0074] The preprocessing module is configured to perform preset processing on the first sequence to generate a second sequence, where the second sequence includes a plurality of token sets classified according to modalities, and the plurality of token sets correspond to a plurality of types of information in the input information in a one-to-one manner.

[0075] The data modeling module is configured to process the second sequence based on an attention mechanism to generate a third sequence, where the third sequence includes bidirectional association information of tokens in each token set in the plurality of token sets, and includes causal association information of tokens between different token sets in the plurality of token sets.

[0076] The action generation module is configured to generate a target action according to the third sequence, where the target action belongs to actions for performing a target task.

[0077] In an optional implementation, after the target action is generated according to the third sequence:

[0078] The action generation module is further configured to add the target action and an incremental image to the input information in a case where the target task is not completed after the target action, where the incremental image is used to reflect environmental information after the target action is performed.

[0079] The action generation module is further configured to trigger the serialization module to return to an operation of generating the first sequence according to the input information.

[0080] In yet another optional implementation, the second sequence includes a text token set, m image token sets, and m-1 action token sets, the text token set corresponds to a task instruction in the input information, an i-th image token set in the m image token sets corresponds to an i-th image in the input information, a j-th action token set in the m-1 action token sets corresponds to a j-th action in the input information, the text token set is a target constraint in the Markov decision process, the i-th image token set is a state at an i-th time in the Markov decision process, and the j-th action token set is an action at a j-th time in the Markov decision process.

[0081] In yet another optional implementation, the data modeling module includes an attention processing module and an action query module, and in terms of processing the second sequence based on an attention mechanism to generate a third sequence:

[0082] The attention processing module is configured to process the second sequence by a causal attention mechanism and a bidirectional attention mechanism to generate a fifth sequence.

[0083] The action query module is configured to add a set of action query parameters after each token set corresponding to the image in the input information in the fifth sequence to generate a third sequence, wherein the set of action query parameters is used to query the relationship between the token sets and actions.

[0084] In yet another optional implementation, in terms of generating the target action according to the third sequence, the action generation module is specifically configured to:

[0085] perform autoregressive processing on the third sequence based on a transformer network to generate the target action.

[0086] In yet another optional implementation, the target action includes n action tokens, the set of action query parameters includes n action query parameters, and the kth action query parameter in the last set of action query parameters in the third sequence is used to extract relevant information of the kth action token in the n action tokens in the third sequence, n is a preset action dimension number, k is 1, 2, …, n, and n is a positive integer.

[0087] In a fourth aspect, the embodiments of the present application provide a model training apparatus, which comprises a serialization module, a preprocessing module, a data modeling module, an action generation module and a parameter optimization module, wherein:

[0088] The serialization module is configured to generate a plurality of first training sequences according to a plurality of training samples, wherein the plurality of training samples include a first positive sample, the first positive sample includes a task instruction, an initial image and a reference action, the task instruction is used to instruct to perform a target task, the initial image is used to reflect environment information corresponding to performing the target task, and the reference action is a standard action of performing the target task;

[0089] The preprocessing module is configured to perform a preset processing on the first training sequence to generate a second training sequence, wherein the second training sequence includes a plurality of token sets classified according to modalities, and the plurality of token sets one-to-one correspond to a plurality of types of information in the training sample;

[0090] The data modeling module is configured to process the second training sequence based on an attention mechanism to generate a third training sequence, wherein the third training sequence includes bidirectional association information of tokens within each token set in the plurality of token sets, and includes causal association information of tokens between different token sets in the plurality of token sets;

[0091] The action generation module is configured to generate a predicted action according to the third training sequence;

[0092] The parameter optimization module is configured to optimize parameters in the action generation model according to the reference action and the predicted action, wherein the action generation model is configured to generate an action for performing a target task according to input information, and the input information comprises a task instruction and an initial image.

[0093] In an optional implementation, the plurality of training samples comprises a second positive sample, and the second positive sample comprises a task instruction, an initial image, and an action obtained by performing noise processing on the reference action.

[0094] In another optional implementation, the plurality of training samples further comprises one or more of a first negative sample, a second negative sample, and a third negative sample, wherein:

[0095] The first negative sample comprises an interference instruction, an initial image, and a reference action, and the interference instruction is configured to instruct to perform a task different from the target task.

[0096] The second negative sample comprises a task instruction, an interference image, and a reference action, and the interference image is configured to reflect environmental information corresponding to performing a task different from the target task.

[0097] The third negative sample comprises a task instruction, an initial image, and an interference action, and the interference action is a standard action corresponding to performing a task different from the target task.

[0098] In yet another optional implementation, the first positive sample further comprises m-1 incremental images, the reference action comprises m reference actions, and the predicted action comprises m predicted actions, each of the m-1 incremental images is configured to reflect environmental information after performing one of the m reference actions, and the m predicted actions correspond to the m reference actions one by one.

[0099] In yet another optional implementation, if the m reference actions are sequentially performed, the target task is completed.

[0100] In yet another optional implementation, in the aspect of performing the preset processing on the first training sequence to generate the second training sequence, the preprocessing module is specifically configured to:

[0101] perform processing on the first training sequence based on a Markov decision process to generate the second training sequence.

[0102] In yet another optional implementation, the second training sequence includes a set of text tokens, m sets of image tokens, m sets of action tokens, the set of text tokens corresponds to the task instruction in the first positive sample, the i-th set of image tokens in the m sets of image tokens corresponds to the i-th image in the first positive sample, the i-th set of action tokens in the m sets of action tokens corresponds to the i-th action in the first positive sample, the set of text tokens is a corresponding target constraint in a Markov decision process, the i-th set of image tokens is a state at the i-th moment in the Markov decision process, and the i-th set of action tokens is an action at the i-th moment in the Markov decision process.

[0103] In yet another optional implementation, the data modeling module includes an attention processing module and an action query module, and in processing the second training sequence based on an attention mechanism to generate a third training sequence:

[0104] The attention processing module is configured to process the second training sequence through a causal attention mechanism and a bidirectional attention mechanism to generate a fifth training sequence.

[0105] The action query module is configured to add a set of action query parameters to each token set corresponding to an image in the first positive sample in the fifth training sequence to generate the third training sequence, wherein the set of action query parameters is used to query the relationship between the plurality of token sets and actions.

[0106] In yet another optional implementation, in generating the predicted action according to the third training sequence, the action generation module is specifically configured to:

[0107] perform autoregressive processing on the third training sequence based on a transformer network to generate the predicted action.

[0108] In yet another optional implementation, the predicted action includes m predicted actions, each of the m predicted actions includes n action tokens, the n action tokens correspond to actions of the predicted action in a plurality of action dimensions respectively, the set of action query parameters includes n action query parameters, the k-th action query parameter in the i-th set of action query parameters in the third training sequence is used to extract related information of the k-th action token in the n action tokens of the i-th sample action in the m sample actions in the third sequence, n is a preset number of action dimensions, i takes 1, 2, …, m, k takes 1, 2, …, n, and m and n are positive integers.

[0109] In a fifth aspect, an embodiment of the present application provides an action generation device, the action generation device comprising a memory and a processor, the memory being configured to store a computer program, and the processor being configured to invoke the computer program to implement the method described in the first aspect or any possible implementation manner of the first aspect.

[0110] In a sixth aspect, an embodiment of the present application provides a model training device, the action generation device comprising a memory and a processor, the memory being configured to store a computer program, and the processor being configured to invoke the computer program to implement the method described in the second aspect or any possible implementation manner of the second aspect.

[0111] In a seventh aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium being configured to store a computer program, and the computer program being configured to implement the method of the first aspect or any possible implementation manner of the first aspect when executed.

[0112] In an eighth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium being configured to store a computer program, and the computer program being configured to implement the method of the second aspect or any possible implementation manner of the second aspect when executed.

[0113] The method, system and device provided by any possible implementation manner of the third aspect, the fifth aspect and the seventh aspect of the present application have beneficial effects which can refer to the beneficial effects of the technical solutions provided by the first aspect and any possible implementation manner of the first aspect, and details are not repeated here.

[0114] The method, system and device provided by any possible implementation manner of the fourth aspect, the sixth aspect and the eighth aspect of the present application have beneficial effects which can refer to the beneficial effects of the technical solutions provided by the second aspect and any possible implementation manner of the second aspect, and details are not repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0115] The following describes the drawings used in the embodiments of the present application.

[0116] FIG. 1 is a schematic diagram of an architecture of an action generation system according to an embodiment of the present application;

[0117] FIG. 2 is a schematic diagram of a flow of an action generation method according to an embodiment of the present application;

[0118] FIG. 3 is a schematic diagram of a flow of another action generation method according to an embodiment of the present application;

[0119] FIG. 4 is a schematic diagram of an interaction process according to an embodiment of the present application;

[0120] FIG. 5 is a schematic diagram of a data structure of a sequence according to an embodiment of the present application;

[0121] FIG. 6 is a schematic diagram of an action dimension according to an embodiment of the present application;

[0122] FIG. 7 is a schematic diagram of a data structure of another sequence according to an embodiment of the present application;

[0123] FIG. 8 is a schematic diagram of a model training method according to an embodiment of the present application;

[0124] FIG. 9 is a schematic diagram of a data structure of yet another sequence according to an embodiment of the present application;

[0125] FIG. 10a is a schematic diagram of a structure of an action generation apparatus according to an embodiment of the present application;

[0126] FIG. 10b is a schematic diagram of a structure of yet another action generation apparatus according to an embodiment of the present application;

[0127] FIG. 11 is a schematic diagram of a structure of a model training apparatus according to an embodiment of the present application;

[0128] FIG. 12 is a schematic diagram of a structure of a computing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0129] The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0130] For ease of understanding, the following exemplary descriptions are given for some concepts related to the embodiments of the present application. As follows:

[0131] 1. Token: In the field of deep learning, a token represents a smallest unit of data in a data processing process. For example, for text data, a text token usually represents a word, a punctuation mark, a letter, or a number in the text, etc. For image data, an image token usually represents a sub-region (also known as a patch) of an image. For action data, an action token usually represents a sub-action of an action with physical meaning in an action dimension.

[0132] 2. Modality: Refers to the source or form of information. Exemplarily, the modality of information can be text, image, video, audio, etc.

[0133] 3. Causal attention mechanism: A causal attention mechanism is an attention mechanism that can maintain a causal relationship in time and allow the model to decide the focus point according to the current and past context when processing sequence data.

[0134] 4. Bidirectional attention mechanism: A bidirectional attention mechanism is a mechanism that considers both forward and backward contextual information when processing sequential data, thereby providing a more comprehensive understanding of the importance of each position in the sequence.

[0135] The above explanations of technical terms can be used in the embodiments below.

[0136] Referring to FIG. 1, FIG. 1 is a schematic diagram of an architecture of an action generation system provided by an embodiment of the present application. The action generation system 10 at least includes a model training device 101, one or more data acquisition devices 102, and one or more model using devices 103.

[0137] The model training device 101 and the data acquisition device 102 can communicate through wired or wireless means. The data acquisition device 102 can send the acquired multiple training samples to the model training device 101. Each training sample includes task instruction information, image information, and reference action (i.e., standard action) information. Correspondingly, the model training device 101 trains an action generation model through the received multiple training samples.

[0138] The model training device 101 and the model using device 103 can communicate through wired or wireless means. Therefore, the model training device 101 can send the trained action generation model (or network) to the model using device 103. Correspondingly, the model using device 103 generates the action required to perform the target task through the received model and performs subsequent processing according to the action.

[0139] Optionally, the data acquisition device 102 includes multiple data acquisition devices, such as a first data acquisition device, a second data acquisition device, and a third data acquisition device. The first data acquisition device is used to acquire the task instruction information in the training sample, the second data acquisition device is used to acquire the image information in the training sample, and the third data acquisition device is used to acquire the reference action information in the training sample. For example, the first data acquisition device is a device capable of voice recognition, the second data acquisition device is a device with a shooting function, and the third data acquisition device can be a device that acquires network data or database data through a network. The third data acquisition device can be a cloud device or a terminal device.

[0140] Optionally, there is only one data acquisition device 102, in which case the data acquisition device 102 can acquire multiple training samples.

[0141] Optionally, the plurality of training samples can have a sequence when performing model training, the model training device 101 can be used only for performing processing tasks related to model training, and the timing of inputting the plurality of training samples into the model training device 101 for model training is controlled by the data acquisition device 102. Optionally, the plurality of training samples are simultaneously trained, and the data acquisition device 102 can simultaneously input the plurality of training samples into the model training device 101 for model training.

[0142] Optionally, the action generation model in the model training device 101 is a model constructed based on a transformer architecture.

[0143] Optionally, the model training device 101 can be a device with strong computing power, for example, a server or a server cluster composed of multiple servers.

[0144] In an optional embodiment, the action generation system 10 can further include an application server. In the training phase, the model training device 101 trains the action generation model using the plurality of samples. In the model using phase, the trained action generation model is stored in the application server. Optionally, the application server can send the trained action generation model to the model using device 103, and the model using device 103 uses the action generation model to process the acquired task instruction type information and image type information, generates an action required to perform a target task, and performs a subsequent processing process according to the action. Alternatively, the model using device 103 sends the acquired task instruction type information and image type information to the application server, the application server uses the action generation model to process the acquired information, generates an action required to perform a target task, and sends the action to the model using device 103, and the model using device 103 performs a subsequent processing process according to the action.

[0145] Optionally, the model using device 103 can be a device capable of performing actions or a component in the device, and the device can be a physical entity (such as a smart robot, a smart mechanical arm, etc.).

[0146] Optionally, the subsequent processing process can be that the model using device 103 controls the device or component capable of performing actions to complete corresponding actions based on the generated action.

[0147] The target task is a task that needs to be completed after one or more actions are performed. Optionally, the target task can be a task in a desktop level control scene (such as placing object A to location B), or a task in a navigation movement scene (such as finding object C).

[0148] Optionally, the model using device 103 can feed the result generated based on the model to the model training device 101, so that the model training device 101 can further train the model based on the result generated by the model using device 103, and the retrained model can be sent to the model using device 103 to update the original model.

[0149] It can be understood that the model training device 101 and the model using device 103 described above can be two separate devices, or can be one device, for example, the above-mentioned embodied entity. When the model training device 101 and the model using device 103 are one device, the communication connection and information interaction process between the two devices mentioned above do not exist. The model training device 101 and the application server described above can be two separate devices, or can be one device, and the present application does not limit this.

[0150] Please refer to FIG. 2, which is a flowchart of an action generation method provided by an embodiment of the present application, which describes the principle of an action generation model. The method can be implemented based on the architecture shown in FIG. 1, or can be implemented based on other architectures. The method includes but is not limited to the following steps:

[0151] Step S201: The model using device generates a first token sequence according to input information. The input information includes an image indicating an execution target task and containing environment information, and the first token sequence includes a plurality of text tokens and a plurality of image tokens. Step S202: The model using device processes the first token sequence based on a causal attention mechanism to generate a second token sequence. The second token sequence includes causal relationship information between a plurality of tokens. Step S203: The model using device performs autoregressive processing on the second token sequence based on a transformer network to generate a first action token. The first action token is a sub-action of an action related to the execution target task in a first action dimension. In addition, the model using device can also add the action token and the image of the environment information of the embodied entity after executing the sub-action to the original input information (i.e. task instruction and initial image), and re-execute the operation of generating the first token sequence to output a second action token. The second action token is a sub-action of the action in a second action dimension. Iteration is performed in this way until the last action token related to the execution target task is generated.

[0152] It can be seen that the method can serialize the task instruction, the image and / or the action, and then perform causal attention mechanism and autoregressive processing on the token sequence, so as to gradually generate sub-actions in a single action dimension to complete the task instruction. However, the causal correlation of the content of the information of a single modality in the input information is weak or non-existent, and the method adopts the causal attention mechanism processing mode for all tokens in the first token sequence, which hinders the information flow between tokens belonging to the same modality, thereby affecting the accuracy of the generated action. Moreover, the method gradually generates sub-actions of an action in a single action dimension, and the time for generating an action is in a linear relationship with the dimension of the action. For a complex task instruction, the method needs to generate multiple actions and the dimension of the action is larger, thereby affecting the efficiency of action generation.

[0153] Referring to FIG. 3, FIG. 3 is a flowchart of another action generation method provided by an embodiment of the present application. The method can be implemented based on the architecture shown in FIG. 1, or can be implemented based on other architectures. The method includes but is not limited to the following steps:

[0154] In step S301, a model using device generates a first sequence according to input information.

[0155] The input information includes multiple types of information. Specifically, the input information includes a task instruction and an initial image, and can also include other information. The task instruction is used to indicate a target task to be executed. It can be understood that the task instruction and the initial image are two types of information. Optionally, the task instruction can be a literal instruction, or a literal instruction generated by recognizing a non-literal instruction. The non-literal instruction can be a voice instruction, an action instruction, and a facial expression instruction, etc. For example, the action instruction can be a body movement (such as waving hands, shaking head, etc.), and the facial expression instruction can be a facial expression (such as smiling, winking, etc.).

[0156] The target task is a task that needs to be completed after one or more actions are performed. Optionally, the target task can be a task in a desktop-level control scene (such as placing object A at location B), or a task in a navigation movement scene (such as finding object C). The initial image is used to reflect initial environmental information corresponding to the execution of the target task, i.e., to reflect the initial environmental state before the execution of the target task. It can be understood that the multiple types of information (such as the task instruction and the initial image) in the input information belong to information of different modalities in the embodied intelligence scene.

[0157] The input information can be collected by the model using device itself or collected by other devices and sent to the model using device. Optionally, the model using device can be a smart robot, the task instruction in the input information can be a text instruction obtained by voice recognition of a voice instruction collected by the smart robot through a configured microphone, and the initial image can be an image containing surrounding environment information photographed by the smart robot through one or more configured cameras after receiving the task instruction. It can be understood that the initial image can be one or more images. For example, if the task instruction is "put the apple into the basket", the initial image can be an image showing "there are an apple and a basket on the table".

[0158] The model using device encodes the input information to map the input information into a first sequence recognizable by a computer. Specifically, the model using device can use a language encoding model and an image encoding model to process the task instruction and the initial image respectively to generate a plurality of text tokens corresponding to the task instruction and a plurality of initial image tokens corresponding to the initial image. Then, the model using device groups the plurality of text tokens and the plurality of image tokens into the first sequence. Optionally, the language encoding model can be a text-to-text transfer transformer (T5), an eXtreme language network (XLNet), a bidirectional encoder representation from transformers (BERT), an attention with linear biases (ALiBi), etc. Optionally, the image encoding model can be a residual network (ResNet), a vision transformer (ViT), etc. It can be understood that each token in the first sequence is represented as a vector or a matrix with the same dimension.

[0159] Step S302: The model using device performs a preset processing on the first sequence to generate a second sequence.

[0160] The second sequence includes multiple token sets categorized by modality. Each of these token sets corresponds one-to-one with various pieces of information in the input information. That is, each token set describes one type of information among the various pieces of information in the input information. It can be understood that these multiple token sets include a set of text tokens describing task instructions and a set of initial image tokens describing the initial image. Optionally, these multiple token sets can be arranged according to the interaction relationships between the various pieces of information in the input information. The interaction relationships in this embodiment can be understood as the causal relationships between various pieces of information during the interaction between the model training device and the external environment in an embodied intelligence scenario, i.e., the relationship between generating actions based on task instructions and the initial image. For example, initial image information is collected only after task instructions are obtained, and an action is generated only based on the task instructions and the initial image.

[0161] In one alternative implementation, the model uses a device to process the first sequence based on a Markov decision process to generate a second sequence. A Markov decision process is a time-dependent, ongoing process and a mathematical framework for describing sequential decision problems, which can be composed of a quintuple (state space, action space, discount factor, reward function, and state transition function).

[0162] The process by which the model training device interacts with the external environment (hereinafter referred to as the "interaction process" for ease of description) can be described using a Markov decision process. For example, please refer to Figure 4, which is a schematic diagram of an interaction process provided in an embodiment of this application. As shown in Figure 4, the model uses the device to determine the state S of the environment at time t. t The action A to be taken at time t t The environment will change according to action A t Give feedback R t+1 (i.e., reward R) t+1 ), and because of the action A t The generation of the environment is caused by state S. t It will transition to state S at time t+1. t+1 This process continues until the target constraint is met (e.g., completing a target task). It can be understood that state S... t and state S t+1 Belongs to the state space, action A t It belongs to the action space.

[0163] The model using device can classify and arrange the plurality of tokens in the first sequence by means of the mathematical framework of Markov decision process. Specifically, the model training device classifies the text tokens and the plurality of initial image tokens in the first sequence into a text token set and an initial image token set according to the modalities, then takes the text token set as a corresponding target constraint in the Markov decision process, takes the initial image token set as a state at a first time in the Markov decision process, and obtains a second sequence.

[0164] For example, referring to FIG. 5, which is a schematic diagram of a data structure of a sequence according to an embodiment of the present application. As shown in FIG. 5, the arrangement order of the tokens in the first sequence B can be represented as: B = {p1, p2, p3, s11, s12}

[0165] The arrangement order of the token sets in the second sequence D can be represented as: D = {P, S1} P = {p1, p2, p3} S1 = {s11, s12}

[0166] Wherein, p1, p2 and p3 are respectively a plurality of text tokens corresponding to the task instruction, and s11 and s12 are respectively a plurality of initial image tokens corresponding to the initial image. P is a text token set in the second sequence D, and S1 is an initial image token set in the second sequence D, which can also be understood as a state at a first time in the Markov decision process.

[0167] The number of text tokens and image tokens in the first sequence in the above example is only an example. In actual application, the number of text tokens is related to the length of the instruction and the language encoding model used, and the number of image tokens is related to the size of the image, the number of images and the image encoding model used.

[0168] Optionally, the first sequence can further include a plurality of action tokens, and the second sequence can further include an action token set corresponding to the initial image token set. Based on the Markov decision process, the model training device can arrange the action token set as an action at a first time in the Markov decision process, behind the initial image token set S1. This is further explained in the subsequent step S304, which is not described in detail here.

[0169] Step S303: The model using device processes the second sequence based on the attention mechanism to generate a third sequence.

[0170] The third sequence includes bidirectional association information of tokens in each token set of the second sequence, and includes causal association information of tokens between different token sets of the plurality of token sets.

[0171] As can be known from step S202, each token set contains a plurality of tokens of the same modality and corresponds to one kind of information in the input information. Considering that the plurality of information in the input information (such as the task instruction and the initial image) belongs to information of different modalities in the embodied intelligence scene, and the information of different modalities has strong causal association, the information of a single modality itself has strong bidirectional association.

[0172] For example, the task instruction in the input information is "put the apple into the basket", and the initial image is an image showing "there are an apple and a basket on the table". It can be understood that the task instruction involves the apple and the basket, and then the initial image containing the environmental information should be an image containing the apple and the basket, that is, the task instruction and the initial image have strong causal association. The bidirectional association between each word in the task instruction is strong, such as "apple" is easily associated with "fruit", or "fruit" is easily associated with "apple". Similarly, the bidirectional association between each sub-region in the initial image is strong, such as the plurality of sub-regions in the initial image form a complete apple image together, and one sub-region (part of the apple image) is easily associated with other sub-regions.

[0173] Therefore, the embodiment of the present application classifies the input information into a plurality of token sets through steps S201 and S203, and then processes the plurality of token sets through the attention mechanism to extract the causal relationship between the information of different modalities and the bidirectional association of the content of the information of a single modality, so as to facilitate subsequent generation of the target action based on the third sequence containing these relationships, thereby avoiding the jump and obvious distortion of the action, and further improving the accuracy of the generated action.

[0174] In an optional embodiment, the model using device can perform bidirectional attention association on the plurality of tokens in each token set in the second sequence through the bidirectional attention mechanism, and perform causal attention association on the plurality of token sets in the second sequence through the causal attention mechanism. Specifically, the model using device processes the second sequence through the causal attention mechanism and the bidirectional attention mechanism to generate a fifth sequence. The embodiment of the present application does not limit the order of processing of the causal attention mechanism and the bidirectional attention mechanism. For example, as shown in FIG. 5, the second sequence D is first processed through the bidirectional attention mechanism to generate a sixth sequence F. The sixth sequence F can be represented as: F={P′,S′1} P′={p1′,p2′,p3′} S′1={s11′,s12′}

[0175] wherein P' is the token set of P in the second sequence D after being processed by the bidirectional attention mechanism. S'1 is the image token set of S1 in the second sequence D after being processed by the bidirectional attention mechanism. p1', p2' and p3' are the plurality of text tokens in P', and s11' and s12' are the plurality of image tokens in S'1. The plurality of tokens in each token set in the sixth sequence F have bidirectional association relationship (indicated by solid arrows in the figure).

[0176] After that, the sixth sequence F is processed by the causal attention mechanism to generate the fifth sequence H. The fifth sequence H can be represented as: H = {P", S"1} P" = {p1", p2", p3"} S"1 = {s11", s12"}

[0177] wherein P" is the token set of P' after being processed by the causal attention mechanism. S"1 is the image token set of S'1 after being processed by the causal attention mechanism. p1", p2" and p3" are the plurality of text tokens in P", and s11" and s12" are the plurality of image tokens in S"1. The plurality of tokens in each token set in the fifth sequence H have causal association relationship (indicated by dashed arrows in the figure).

[0178] As can be seen, the plurality of token sets in the fifth sequence can also be considered to correspond one-to-one to the plurality of information in the input information.

[0179] After that, the model using device adds a set of action query parameters to each token set corresponding to the image in the input information in the fifth sequence to generate a third sequence. The set of action query parameters is used to query the relationship between the plurality of token sets in the third sequence and the action.

[0180] Specifically, the set of action query parameters includes n action query parameters. For example, as shown in FIG. 5, the set of action query parameters Q includes 3 action query parameters, and the image token set S"1 in the fifth sequence H generates a third sequence R after being added with the set of action query parameters Q. The third sequence R can be represented as: R = {P", S"1, Q} Q = {q1, q2, q3}

[0181] wherein q1, q2 and q3 are the plurality of action query parameters in the set of action query parameters Q. Each action query parameter is a vector or a matrix with the same dimension as the text token and the image token in the third sequence.

[0182] And n is equal to a preset action dimension number. As described above, one action with physical meaning can be decomposed into sub-actions in n action dimensions. The kth action query parameter in the action query parameter set is used to extract relevant information between the multiple token sets in the third sequence and a sub-action of one action in the kth action dimension. k takes 1, 2, …, n, and n is a positive integer. The subsequent step S304 has a further explanation, which is not described here.

[0183] The embodiment of the present application does not strictly limit the action dimension number n, which can be set according to the actual application scenario. Optionally, the model using device can be a smart robot, and the mechanical arm of the robot is usually the part that performs the action in the desktop level control scene. The action dimension of the mechanical arm can be 7. As shown in FIG. 6, in a three-dimensional Cartesian coordinate system, the 8 action dimensions of the mechanical arm 50 can be: the position in the x-axis direction, the position in the y-axis direction, the position in the z-axis direction, the rotation angle θ pitch in the x-axis direction, the rotation angle θ yaw in the y-axis direction, the rotation angle θ roll in the z-axis direction, and the clamping opening angle θ open or the opening degree of the clamp of the mechanical arm 50. The judgment standard and representation method of the opening degree can be preset.

[0184] Yet another optional, the model using device can be a smart robot, and the whole robot is the part that performs the action in the navigation movement scene. The action dimension of the robot can be 3. In a two-dimensional Cartesian coordinate system, the 3 action dimensions of the robot can be: the position in the x-axis direction, the position in the y-axis direction, and the rotation angle of the robot in the x-y plane.

[0185] In addition, as described above, the second sequence can also include an action token set corresponding to the initial image token set, and then the third sequence can also include an action token set processed by the bidirectional attention mechanism from the action token set.

[0186] As shown in FIG. 5, the modal classification and the ordering according to the interaction relationship of the multiple tokens in the first sequence in step S202 are beneficial to the processing of the mixed attention mechanism (i.e., the bidirectional attention mechanism and the causal attention mechanism) in step S203 and the embedding of the action query parameter set, generating the third sequence. Compared with the method of directly processing the multiple tokens in the first sequence based on the causal attention mechanism, the embodiment of the present application increases the information flow between the multiple tokens in the same mode. The number of tokens that increase the information flow can be represented by the following formula:

[0187] Wherein, L, M and N are the number of text tokens in the second sequence, the number of image tokens and the number of action tokens respectively, and W is the time length of the second sequence.

[0188] It should be noted that the time length of the second sequence can be understood as the number of time points corresponding to the token set in the second sequence. For example, as described above, the second sequence can be obtained by the model using device processing the first sequence based on the Markov decision process. When the second sequence only contains the text token set P and the initial image set S1, the initial image set S1 can be regarded as the state at the first time point in the Markov decision process, and then W is 1.

[0189] In an optional embodiment, the model using device can first add an action query parameter set to each token set corresponding to the image in the input information in the second sequence, and then process it based on the causal attention mechanism and the bidirectional attention mechanism to generate a third sequence. That is, the order of the two processing steps of the attention mechanism processing and the embedding of the action query parameter set is not limited when the model using device processes the second sequence.

[0190] Step S304: The model using device generates a target action according to the third sequence.

[0191] In an optional embodiment, the model using device can perform autoregressive processing on the third sequence based on a transformer network to generate a target action. The target action is an action for performing a target task. The target action includes n action tokens, and the n action tokens correspond to the actions of the target action in n action dimensions respectively. For the explanation of the number n of action dimensions, please refer to the corresponding part of the explanation in step S303, which will not be repeated here.

[0192] The transformer network generally consists of an encoder and a decoder. The encoder is responsible for mapping the input sequence (such as the third sequence) to a representation in a high-dimensional space, while the decoder maps this representation back to the output sequence (such as the target action). Each encoder and decoder contains a self-attention mechanism and a feed-forward neural network (FFN). The self-attention network is used to calculate the correlation between each element and other elements in the input sequence to construct an attention weight matrix. The FFN network is used to perform nonlinear transformation and mapping on the feature data by using the autoregressive mechanism while maintaining the position information.

[0193] Optionally, the third sequence includes a set of action query parameters, and the model uses the device to extract, based on the transformer network, relevant information between the third sequence and the kth action token in the n action tokens by using an action query parameter in the set of action query parameters when generating the target action, and then map the third sequence to the target action. k is 1, 2, …, n.

[0194] For example, as shown in FIG. 5, the target action a1 can be represented as: a1={a11,a12,a13}

[0195] wherein a11, a12 and a13 are respectively a plurality of action tokens in the target action a1.

[0196] The transformer network can extract, by using the first action query parameter q1 in the set of action query parameters Q, relevant information between all tokens in the third sequence R arranged before q1 and the first action token (i.e., a11) in the target action a1, to generate a1. Similarly, the transformer network generates the second action token (i.e., a12) by using the second action query parameter q2. The transformer network generates the third action token (i.e., a13) by using the third action query parameter q3.

[0197] That is, the embodiments of the present application can extract information of multiple modalities in the input information that are most relevant to the sub-actions in the target action in multiple action dimensions, respectively, not only can enhance the flow of information that is beneficial to the generation of subsequent target actions, further improve the accuracy of the generated action, but also can realize the display alignment between the information of multiple modalities in the input information and the sub-actions in different dimensions, and synchronize the generation of multiple action tokens in a target action, thereby improving the efficiency of action generation.

[0198] Further, after generating the target action, the model using device can execute the target action to complete the target task indicated by the task instruction.

[0199] In an optional implementation, the target task is a task that needs to be completed by performing multiple actions, and the model using device can add the target action and the environmental information after executing the target action to the existing input information in the case that the target task is not completed after executing one target action, and then execute the operation of generating a new target action again (i.e., repeat steps S301 to S304) based on the new input information, using this iterative way until the target task is completed.

[0200] Specifically, the model using device can acquire an incremental image in a case where the target task is not completed after the target action is performed. The incremental image is used to reflect the environmental information after the target action is performed, and can be collected by the model using device itself or collected by other devices and sent to the model using device. Optionally, the model using device can be a smart robot, and the incremental image can be an image containing environmental information taken by one or more cameras configured on the smart robot in a case where the smart robot confirms that the target task is not completed after the target action is performed. It can be understood that the incremental image can be one or more images. For example, if the task instruction is “put the apple into the basket”, the incremental image can be an image showing that the mechanical arm of the robot touches the apple. Optionally, if one of the plurality of action tokens in the target action is a preset action token, the model using device can determine that the target task is not completed after the target action is performed. That is, the value of one of the plurality of action tokens can be used as a basis for the model using device to determine whether the target task is completed. The specific position of the action token in the plurality of action tokens and the value of the action token can be set in the model training stage.

[0201] Then, the model using device adds the target action and the incremental image to the original input information (containing the target instruction and the initial image), that is, the updated input information contains four kinds of information: the target instruction, the initial image, the target action and the incremental image, and the operations of steps S301 to S304 are repeated for the updated input information to generate an updated target action.

[0202] For ease of understanding, see FIG. 7 for an example. FIG. 7 is a schematic diagram of another sequence of data structures provided by an embodiment of the present application. As shown in FIG. 7, the input information includes a target instruction, an initial image, a target action and an incremental image. As known from step S301, the first sequence B can be represented as: B = {p1, p2, p3, s11, s12, a11, a12, a13, s21, s22}

[0203] wherein a11, a12 and a13 are respectively a plurality of action tokens in the target action, and s21 and s22 are respectively a plurality of incremental image tokens corresponding to the incremental image. The meanings of other elements in the first sequence B can be referred to the specific description of the first sequence B in step S302, which will not be repeated here.

[0204] As known from step S302, the second sequence D can be represented as: D = {P, S1, A1, S2} P = {p1, p2, p3}, S1 = {s11, s12}, A1 = {a11, a12, a13}, S2 = {s21, s22}

[0205] A1 is the action token set in the second sequence D, which can also be understood as the action at the first time in the Markov decision process. S2 is the incremental image token set in the second sequence D, which can also be understood as the state at the second time in the Markov decision process. The meanings of other elements in the second sequence D can be referred to the specific description of the second sequence D in step S302, which will not be repeated here.

[0206] As can be seen from step S303, the third sequence R can be represented as: R = {P", S"1, Q, A"1, S"2, Q} P = {p1", p2", p3"} S1 = {s11", s12"} A1 = {a11", a12", a13"} S2 = {s21", s22"} Q = {q1, q2, q3}

[0207] A"1 is the action token set A1 processed by the mixed attention mechanism, and S"1 is the image token set S1 processed by the mixed attention mechanism. a11" and a12" are respectively a plurality of image tokens in A"1, and s21" and s22" are respectively a plurality of image tokens in S"2. The information flow between the multiple tokens in each token set during the mixed attention mechanism processing process is shown by solid arrows, and the information flow between the token sets is shown by dashed arrows. The meanings of other elements in the third sequence R can be referred to the specific description of the third sequence R shown in FIG. 5 in step S303, which will not be repeated here.

[0208] As can be seen from step S304, based on the transformer network, the first action query parameter q1 in the parameter set Q can be used to extract the relevant information between all tokens arranged in front of q1 in the third sequence R and the first action token (i.e., a21) in the new target action a2, to generate a21. Similarly, the transformer network generates the second action token (i.e., a22) through the second action query parameter q2. The transformer network generates the third action token (i.e., a23) through the third action query parameter q3.

[0209] It can be understood that for a target task that needs to be completed by performing m target actions, the model using device can perform the operations of steps S301 to S304 m times to generate m target actions step by step. The generation of each new target action is based on the previously generated all target actions and the environmental information after the execution of all target actions as input information, so the embodiments of the present application can ensure the relevance and continuity between the previous and subsequent actions, thereby ensuring the accuracy of each action.

[0210] Based on the detailed explanations of steps S301-S304 above and Figure 7, the input information in this embodiment includes task instructions, m images (i.e., the initial image and m-1 incremental images), and m-1 target actions. The second sequence includes a set of text tokens, a set of m image tokens, and a set of m-1 action tokens. The text token set corresponds to the task instructions in the input information. The i-th image token set in the m image token set corresponds to the i-th image in the input information. The j-th action token set in the m-1 action token set corresponds to the j-th target action in the input information. The text token set represents the target constraint corresponding to the Markov decision process. The i-th image token set represents the state at the i-th moment in the Markov decision process. The j-th action token set represents the action at the j-th moment in the Markov decision process.

[0211] For example, the set of tokens in the second sequence D is arranged in the following order: B = {P, S1, A1, S2, A2, ..., S} m-1 A m-1 ,S m}

[0212] Where P is the set of text tokens, S i Let A be the set of the i-th image tokens. j Let i be the set of action tokens for the j-th action, where i is 1, 2, ..., m, and j is 1, 2, ..., m-1, where m is a positive integer.

[0213] Furthermore, the sets in the third sequence R are ordered as follows: R = {P″,S″1,Q,A″1,S″2,Q,A″2,…,S″} m-1 Q,A″ m-1 ,Q,S″ m}

[0214] Where P″ is the set of text tokens after P has been processed by the hybrid attention mechanism, and S″ i For S i The set of image tokens, A″, processed by the hybrid attention mechanism j For A j The set of action tokens processed by the hybrid attention mechanism.

[0215] In summary, the embodiment of the present application can classify the input information by modalities, process multiple token sets corresponding to multiple types of information in the input information through an attention mechanism to extract the causal relationship between different modal information and the bidirectional association relationship of the content of a single modality information, and then generate a target action based on a third sequence containing these relationships, thereby avoiding jumps and obvious distortions in the action, and further improving the accuracy of the generated action. In addition, the embodiment of the present application can extract information of multiple modalities that are most relevant to sub-actions of the target action in multiple action dimensions in the input information through multiple action query parameters, which not only enhances the flow of information beneficial to the generation of subsequent target actions, further improves the accuracy of the generated action, but also achieves explicit alignment between multiple modalities of information in the input information and sub-actions in different dimensions, and synchronously generates multiple action tokens in a target action, thereby improving the efficiency of action generation.

[0216] As can be seen from the embodiments shown in FIGS. 1 and 3, the model using device can perform the operations described in steps S301 to S304 using the action generation model sent by the model training device. The action generation model can be obtained by the model training device by training multiple training samples. For ease of understanding, please refer to FIG. 8, which is a flowchart of a model training method according to an embodiment of the present application. The method can be implemented based on the architecture shown in FIG. 1, or can be implemented based on other architectures. The method includes but is not limited to the following steps:

[0217] Step S801: The model training device generates multiple first training sequences according to multiple training samples.

[0218] The multiple training samples include positive samples, or the multiple training samples include positive samples and negative samples, and the multiple training samples correspond one-to-one to the multiple first training sequences. In the following, steps S801 to S805 are explained and described taking the example that the multiple training samples include positive samples and negative samples.

[0219] In an optional embodiment, each training sample includes instruction information, m image information and m action information, the positive sample includes a first positive sample and / or a second positive sample, and the negative sample includes one or more of a first negative sample, a second negative sample and a third negative sample. Specifically, the first positive sample includes a task instruction, an initial image, m-1 incremental images and m reference actions, the task instruction is used to indicate a target task, the target task is a task that needs to be completed by performing m actions, the initial image is used to reflect initial environment information corresponding to the execution of the target task, and the incremental image is used to reflect environment information after performing one reference action. For specific explanations of the target task, the initial image and the incremental image, refer to the descriptions of the corresponding parts in steps S301 and S304, which will not be described here in detail. The reference action is a standard action of the target task indicated by the task instruction, which can be pre-set or selected according to experience. The second positive sample includes a task instruction, an initial image, m-1 incremental images and m-1 actions obtained by performing noise processing on the m-1 reference actions, i.e., noise actions. It should be noted that the noise processing in the embodiments of the present application is a processing mode that causes the reference action to be slightly distorted, that is, the similarity or difference between the noise action and the reference action is very high or very small, and the judgment standard of the similarity or difference can be pre-set. It can be understood that the embodiments of the present application can improve the recognition degree of the model to the correct action by introducing multiple positive samples,

[0220] The first negative sample includes an interference instruction, an initial image, m-1 incremental images and m reference actions, the interference instruction is used to indicate a task different from the target task. The second negative sample includes a task instruction, m interference images (including an interference image corresponding to the initial image and m-1 interference images corresponding to the m-1 incremental images) and m reference actions, the interference image corresponding to the initial image is used to reflect initial environment information corresponding to the execution of a task different from the target task, and the interference image corresponding to the incremental image is used to reflect environment information after performing an action different from one reference action. The third negative sample includes a task instruction, an initial image and m interference actions, the interference action is a standard action corresponding to the execution of a task different from the target task. It should be noted that the "task different from the target task" in the embodiments of the present application refers to a task that is significantly different or has a low similarity to the target task, and the judgment standard of the similarity or difference can be pre-set. It can be understood that the embodiments of the present application can process text information, image information and action information according to the modalities of the information to set multiple negative samples, which is beneficial to the model training device to subsequently improve the alignment ability of the action generation model to different modal information by the way of contrast learning based on multiple training samples, thereby improving the accuracy of the generated actions.

[0221] The embodiment of the present application can set positive samples and negative samples based on one task instruction, and the plurality of training samples can include positive samples and negative samples set based on a plurality of task instructions respectively. The embodiment of the present application does not limit the number of positive samples and negative samples corresponding to one task, that is, the embodiment of the present application does not limit the number of the first positive sample, the second positive sample, the first negative sample, the second negative sample and the third negative sample.

[0222] The specific principle of generating the first training sequence in step S801 can refer to the principle of generating the first sequence in step S301 of the embodiment shown in FIG. 3, which will not be described here in detail. For the convenience of understanding, taking the target task as an example, which needs to perform two actions to complete, the data characteristics of the first training sample are further explained and described. Please refer to FIG. 9, which is another data structure diagram of a sequence provided by the embodiment of the present application. As shown in FIG. 9, the first training sequence B 1+ may be expressed as:

[0223] Wherein, p1, p2 and p3 are a plurality of tokens corresponding to the task instruction respectively, s11 and s12 are a plurality of initial image tokens corresponding to the initial image respectively, s21 and s22 are a plurality of incremental image tokens corresponding to the incremental image respectively, and are a plurality of reference action tokens corresponding to the first reference action respectively, and are a plurality of reference action tokens corresponding to the second reference action respectively.

[0224] The first training sequence B 2+ corresponding to the second positive sample can be expressed as:

[0225] Wherein, and are a plurality of noise action tokens corresponding to the first noise action respectively, and are a plurality of noise action tokens corresponding to the second noise action respectively.

[0226] The first training sequence B 1- corresponding to the first negative sample can be expressed as:

[0227] Wherein, p1(-), p2(-) and p3(-) are a plurality of interference tokens corresponding to the interference instruction respectively.

[0228] The first training sequence B 2- corresponding to the second negative sample can be expressed as:

[0229] wherein s11(-) and s12(-) are respectively a plurality of interference image tokens corresponding to the first interference image, and s21(-) and s22(-) are respectively a plurality of interference image tokens corresponding to the second interference image.

[0230] The first training sequence B corresponding to the third negative sample 3- may be represented as:

[0231] wherein, and are respectively a plurality of interference action tokens corresponding to the first interference action, and are respectively a plurality of interference action tokens corresponding to the second interference action.

[0232] Step S802: The model training device performs preset processing on the first training sequence to generate a second training sequence.

[0233] The first training sequence is the first training sequence in step S801, that is, the model training device performs the operations of steps S802 to S805 on each of the plurality of first training sequences in step S801. The principle of generating the second training sequence in step S802 can refer to the principle of generating the second sequence in step S302 of the embodiment shown in FIG. 3, which will not be described in detail here.

[0234] For example, the first training sequence can be the first training sequence B corresponding to the first positive sample 1+ , and the second training sequence D 1+ may be represented as:

[0235] wherein P is a character token set, S1 and S2 are respectively an initial image token set and an incremental image token set, and are respectively a first action token set and a second action token set.

[0236] Step S803: The model training device processes the second training sequence based on an attention mechanism to generate a third training sequence.

[0237] The principle of generating the third training sequence in step S803 can refer to the principle of generating the third sequence in step S303 of the embodiment shown in FIG. 3, which will not be described in detail here.

[0238] For example, the second training sequence D 1+ corresponds to the third training sequence R1+ may be represented as:

[0239] wherein P" is the token set of P processed by the mixed attention mechanism, S"1 is the token set of S1 processed by the mixed attention mechanism, and are respectively and the token set of action processed by the mixed attention mechanism, and Q is the set of action query parameters.

[0240] Step S804: The model training device generates a predicted action according to the third training sequence.

[0241] In an optional implementation, the model training device performs autoregressive processing on the third sequence based on a transformer network to generate m predicted actions. Each predicted action includes n action tokens, and the n action tokens correspond to actions of the predicted action in n action dimensions respectively. In an optional implementation, when the model training device generates a predicted action based on the transformer network, the model training device extracts information related to m predicted actions in the third training sequence through m sets of action query parameters in the third training sequence respectively, to obtain m predicted actions.

[0242] For example, the model training device can generate two predicted actions, i.e., a predicted action β1 and a predicted action β2, according to the third training sequence R 1+ may be represented as: β1={a11,a12,a13} β2={a21,a22,a23}

[0243] wherein a11, a12 and a13 are respectively a plurality of action tokens in the predicted action β1, and a21, a22 and a23 are respectively a plurality of action tokens in the predicted action β2.

[0244] The transformer network can extract relevant information between all tokens arranged in front of the first action query parameter q1 in the first set of action query parameters Q in the third training sequence R 1+ and the first action token (i.e., a11) in the predicted action β1 through the first action query parameter q1 to generate a1. Similarly, the transformer network generates the second action token (i.e., a12) through the second action query parameter q2. The transformer network generates the third action token (i.e., a13) through the third action query parameter q3. Simultaneously, the transformer network can extract relevant information between all tokens arranged in front of the second set of action query parameters Q in the third training sequence R 1+The first action query parameter q1 in the second action query parameter set Q extracts relevant information between all tokens preceding q1 in the third sequence R and the first action token (a21) in the predicted action β2, generating a1. Similarly, the transformer network generates the second action token (a22) using the second action query parameter q2. The transformer network generates the third action token (a23) using the third action query parameter q3.

[0245] For a detailed explanation of the transformer network and the action query parameter set, please refer to the description of the corresponding part in step S304 of the embodiment shown in Figure 3, which will not be elaborated here.

[0246] Step S805: The model training device optimizes the parameters in the action generation model based on the reference action and the predicted action.

[0247] The model training device optimizes the parameters related to generating and predicting actions in the action generation model (such as parameters in the hybrid attention mechanism, parameters in the transformer network, and action query parameters in the action query parameter set) through a loss function.

[0248] In one optional implementation, two positive samples (first positive sample and second positive sample) and three negative samples (first negative sample, second negative sample and third negative sample) are set based on a task instruction. Then, multiple training samples can include 2N positive samples and 3N negative samples set based on N task instructions.

[0249] The model training device compares the m predicted actions corresponding to each of the 2N positive samples in the multiple training samples with the m reference actions corresponding to the task instructions in that positive sample, and calculates the loss value of the 2N positive samples so that the closer the predicted action corresponding to each positive sample is to the reference action, the smaller the loss value.

[0250] For example, the model training device can group the m predicted actions corresponding to each positive sample into a predicted action sequence, and the m reference actions corresponding to that positive sample into a reference action sequence. Next, the model training device can use a softmax function or a sigmoid function to convert the predicted action sequence and the reference action sequence into a target probability sequence and a reference probability sequence, respectively. It can be understood that if the predicted action sequence contains m*n predicted action tokens, and the reference action sequence contains m*n reference action tokens, then both the target probability sequence and the reference probability sequence also contain m*n probability values. Afterwards, the model training device can calculate the cross-entropy loss value L1 for 2N positive samples using the cross-entropy loss function.

[0251] Optionally, the cross-entropy loss function can be expressed as follows:

[0252] where y rg is the gth probability value in the predicted action sequence corresponding to the rth positive sample in the 2N positive samples, is the gth probability value in the reference probability sequence in the reference action sequence corresponding to the rth positive sample, g takes 1, 2, …, m*n, r takes 1, 2, …, 2N, N, m and n all take positive integers.

[0253] Synchronously, since each task instruction in the multiple training samples corresponds to two positive samples, the model training device takes the two positive samples corresponding to each task instruction in the multiple training samples as a positive sample pair, that is, the multiple training samples include N positive sample pairs corresponding to N task instructions and 2N negative samples. It can be understood that the positive samples in each positive sample pair and the other samples in the multiple training samples except the positive sample pair can form negative sample pairs two by two. The model training device can calculate the gap between the positive sample pairs and the gap between the negative sample pairs, so as to calculate the loss value of the multiple training samples, so that the smaller the gap between the predicted actions of the samples in the positive sample pairs and the larger the gap between the predicted actions of the samples in the negative sample pairs, the smaller the loss value.

[0254] For example, the model training device can calculate the contrastive learning loss value L2 of the multiple training samples according to the contrastive learning loss (Info NCE loss) function. Optionally, the Info NCE loss function can be expressed as follows:

[0255] where β r is the last predicted action of the m predicted actions corresponding to the rth positive sample in the 2N positive samples, β r+ is the last predicted action of the m predicted actions corresponding to the positive sample in the same positive sample pair as the rth positive sample, β w is the last predicted action of the m predicted actions corresponding to the wth sample in the multiple training samples. The Sim function is a function for calculating the cosine similarity, T is a temperature coefficient, and r takes 1, 2, …, 2N.

[0256] Then, the model training device calculates the total loss value L of the multiple training samples according to the cross-entropy loss value L1 and the contrastive learning loss value L2. Specifically, the model training device can calculate the total loss value L of the multiple training samples according to the following formula:

[0257] wherein, and are the weighting coefficients of L1 and L2, respectively.

[0258] It can be seen from the above formula that the total loss value is smaller when the predicted action corresponding to each positive sample is closer to the reference action, the difference between the predicted actions of the samples in the positive sample pair is smaller, and the difference between the predicted actions of the samples in the negative sample pair is larger. The model training device can continuously optimize the parameters used by the action generation model in generating actions, such as the parameters in the hybrid attention mechanism, the parameters in the transformer network, and the parameters in the action query parameter set, until the total loss value is smaller than a preset threshold.

[0259] It can be seen from the above formula that the total loss value is smaller when the predicted action corresponding to each positive sample is closer to the reference action, the difference between the predicted actions of the samples in the positive sample pair is smaller, and the difference between the predicted actions of the samples in the negative sample pair is larger. The model training device can continuously optimize the parameters used by the action generation model in generating actions, such as the parameters in the hybrid attention mechanism, the parameters in the transformer network, and the parameters in the action query parameter set, until the total loss value is smaller than a preset threshold.

[0260] It can be seen from the above formula that the total loss value is smaller when the predicted action corresponding to each positive sample is closer to the reference action, the difference between the predicted actions of the samples in the positive sample pair is smaller, and the difference between the predicted actions of the samples in the negative sample pair is larger. The model training device can continuously optimize the parameters used by the action generation model in generating actions, such as the parameters in the hybrid attention mechanism, the parameters in the transformer network, and the parameters in the action query parameter set, until the total loss value is smaller than a preset threshold.

[0261] Further, in order to verify the effects of the hybrid attention mechanism, the action query mechanism, and the contrast learning mechanism in the training process of the action generation model, respectively, the models obtained by the embodiments and various implementation manners (implementation manners 1 to 7) are tested and verified against different difficulty levels of task instruction sets, and the test results are shown in Table 1:

[0262] Table 1 Success rate of target task set of different difficulty levels

[0263] The numerical values in Table 1 are in percentage (%).

[0264] It should be noted that, compared with the embodiments of the present application, the implementation manner 1 is an implementation manner of removing only the negative samples containing the interference instructions in the plurality of training samples in the model training process, the implementation manner 2 is an implementation manner of removing the negative samples containing the interference images in the plurality of training samples, the implementation manner 3 is an implementation manner of removing the negative samples containing the interference reference actions in the plurality of training samples in the model training process, the implementation manner 4 is an implementation manner of removing all the negative samples in the plurality of training samples in the model training process, the implementation manner 5 is an implementation manner of replacing the hybrid attention mechanism with the causal attention mechanism in the model training process, the implementation manner 6 is an implementation manner of removing the action query set in the model training process, and the implementation manner 7 is an implementation manner of replacing the hybrid attention mechanism with the causal attention mechanism and removing the action query set.

[0265] The task instruction set with the difficulty level of “difficult” includes a plurality of difficult task instructions, the task instruction set with the difficulty level of “moderate” includes a plurality of moderate task instructions, the task instruction set with the difficulty level of “easy” includes a plurality of easy task instructions, and the task instruction set with the difficulty level of “average” includes a plurality of difficult task instructions, a plurality of moderate task instructions and a plurality of easy task instructions. Optionally, the difficulty level of the task instruction can be determined according to the number of actions required by the target task indicated by the task instruction and / or the difficulty level of the actions.

[0266] As shown in Table 1, the hybrid attention mechanism, the action query mechanism and the contrast learning mechanism in the embodiments of the present application complement each other, and can jointly improve the accuracy of the generated actions.

[0267] In addition, in order to verify the effect of the action generation model provided in the embodiments of the present application, the embodiments of the present application and the related art shown in FIG. 1 are respectively tested and verified for the target task “placing object A at location B” based on a robot simulation and training platform (Maniskill) simulation environment, and the test results are shown in Table 2:

[0268] Table 2: Success rate under different conditions

[0269] The numerical values in Table 2 are in percentage (%).

[0270] It should be noted that A for visible object means that object A has a corresponding training sample, A for unseen object means that object A has no corresponding training sample, and the interference object refers to an object similar to object A, for example, when object A is an apple, the interference object can be a pear, a peach, etc.

[0271] As shown in Table 2, compared with related technologies, the action generation model provided in this application embodiment improves the success rate by 16.75 percentage points for visible objects, by 20.6 percentage points for unseen objects, and by 11.07 percentage points in environments with interfering objects. In other words, compared with related technologies, the actions generated in this application embodiment have significant improvements in accuracy, generalization, and robustness.

[0272] The following will describe the action generation apparatus provided in the embodiments of this application.

[0273] Figure 10a is a schematic diagram of an action generation device provided in an embodiment of this application. The action generation device 100 can be the model-using device or a component in the model-using device in the above method embodiment. The action generation device 100 may include a serialization module 1001, a preprocessing module 1002, a data modeling module 1003, and an action generation module 1004. The detailed description of each unit is as follows:

[0274] The serialization module 1001 is used to generate a first sequence based on the input information, wherein the input information includes multiple types of information, including task instructions and an initial image. The task instructions are used to instruct the execution of a target task, and the initial image is used to reflect the environmental information corresponding to the execution of the target task.

[0275] The preprocessing module 1002 is used to perform pre-processing on the first sequence to generate a second sequence, wherein the second sequence includes multiple token sets classified according to modality, and the multiple token sets correspond one-to-one with various information in the input information;

[0276] The data modeling module 1003 is used to process the second sequence based on the attention mechanism to generate a third sequence, wherein the third sequence includes bidirectional association information of tokens within each of multiple token sets, and causal association information of tokens between different token sets in multiple token sets.

[0277] The action generation module 1004 is used to generate a target action based on a third sequence, wherein the target action is an action that performs the target task.

[0278] In one alternative implementation, the additional information includes the target action and incremental image, after the target action is generated based on the third sequence:

[0279] The action generation module 1004 is also used to add target action and incremental image to the input information when the target task is not completed after the target action. The incremental image is used to reflect the environmental information after the target action is performed.

[0280] The action generation module 1004 is further configured to trigger the serialization module 1001 to return an operation of generating a first sequence according to the input information.

[0281] In yet another optional implementation, the second sequence includes a token set of text, m token sets of images, and m-1 token sets of actions, the token set of text corresponds to a task instruction in the input information, an i-th token set of images in the m token sets of images corresponds to an i-th image in the input information, a j-th token set of actions in the m-1 token sets of actions corresponds to a j-th action in the input information, the token set of text is a corresponding target constraint in the Markov decision process, the i-th token set of images is a state at an i-th time in the Markov decision process, and the j-th token set of actions is an action at a j-th time in the Markov decision process.

[0282] In yet another optional implementation, as shown in FIG. 10b, the data modeling module 1003 includes an attention processing module 1005 and an action query module 1006, and the attention processing module 1005 is configured to process the second sequence based on an attention mechanism to generate a third sequence.

[0283] The attention processing module 1005 is configured to process the second sequence by a causal attention mechanism and a bidirectional attention mechanism to generate a fifth sequence.

[0284] The action query module 1006 is configured to add an action query parameter set after each token set corresponding to an image in the input information in the fifth sequence to generate the third sequence, and the action query parameter set is used to query a relationship between a plurality of token sets and an action.

[0285] In yet another optional implementation, in generating a target action according to the third sequence, the action generation module 1004 is specifically configured to:

[0286] perform autoregressive processing on the third sequence based on a transformer network to generate the target action.

[0287] In yet another optional implementation, the target action includes n action tokens, the action query parameter set includes n action query parameters, and a k-th action query parameter in a last action query parameter set in the third sequence is used to extract relevant information of a k-th action token in the n action tokens in the third sequence, n is a preset action dimension number, k is 1, 2, …, n, and n is a positive integer.

[0288] FIG. 11 is a structural schematic diagram of a model training apparatus provided by an embodiment of the present application. The model training apparatus 110 can be the model training device in the method embodiments or a device in the model training device. The model training apparatus 110 can include a serialization module 1001, a preprocessing module 1002, a data modeling module 1003, an action generation module 1004, and a parameter optimization module 1007. The detailed descriptions of the respective units are as follows:

[0289] The serialization module 1001 is configured to generate a plurality of first training sequences according to a plurality of training samples. The plurality of training samples include a first positive sample. The first positive sample includes a task instruction, an initial image, and a reference action. The task instruction is used to instruct to perform a target task. The initial image is used to reflect environmental information corresponding to performing the target task. The reference action is a standard action of performing the target task.

[0290] The preprocessing module 1002 is configured to perform a preset processing on the first training sequence to generate a second training sequence. The second training sequence includes a plurality of token sets classified according to modalities. The plurality of token sets correspond to a plurality of types of information in the training sample one by one.

[0291] The data modeling module is configured to process the second training sequence based on an attention mechanism to generate a third training sequence. The third training sequence includes bidirectional association information of tokens in each token set in the plurality of token sets, and includes causal association information of tokens between different token sets in the plurality of token sets.

[0292] The action generation module 1004 is configured to generate a predicted action according to the third training sequence.

[0293] The parameter optimization module 1007 is configured to optimize parameters in an action generation model according to the reference action and the predicted action. The action generation model is used to generate an action of performing the target task according to input information. The input information includes the task instruction and the initial image.

[0294] In an optional implementation, the plurality of training samples include a second positive sample. The second positive sample includes the task instruction, the initial image, and an action obtained by performing noise processing on the reference action.

[0295] In another optional implementation, the plurality of training samples further include one or more of a first negative sample, a second negative sample, and a third negative sample. Wherein:

[0296] The first negative sample includes an interference instruction, the initial image, and the reference action. The interference instruction is used to instruct to perform a task different from the target task.

[0297] The second negative sample includes a task instruction, an interference image, and a reference action, and the interference image is used to reflect environment information corresponding to performing a task different from the target task.

[0298] The third negative sample includes a task instruction, an initial image, and an interference action, and the interference action is a standard action corresponding to performing a task different from the target task.

[0299] In yet another optional implementation, the first positive sample further includes m-1 incremental images, the reference action includes m reference actions, and the predicted action includes m predicted actions, each of the m-1 incremental images is used to reflect environment information after performing one of the m reference actions, and the m predicted actions correspond to the m reference actions one by one.

[0300] In yet another optional implementation, if the m reference actions are sequentially performed, the target task is completed.

[0301] In yet another optional implementation, in the aspect of performing the preset processing on the first training sequence to generate the second training sequence, the preprocessing module 1002 is specifically configured to:

[0302] perform processing on the first training sequence based on a Markov decision process to generate the second training sequence.

[0303] In yet another optional implementation, the second training sequence includes a text token set, m image token sets, and m action token sets, the text token set corresponds to the task instruction in the first positive sample, an i-th image token set in the m image token sets corresponds to the i-th image in the first positive sample, an i-th action token set in the m action token sets corresponds to the i-th action in the first positive sample, the text token set is a target constraint corresponding in the Markov decision process, the i-th image token set is a state at an i-th time in the Markov decision process, and the i-th action token set is an action at the i-th time in the Markov decision process.

[0304] In yet another optional implementation, the data modeling module includes an attention processing module and an action query module, and in the aspect of performing processing on the second training sequence based on an attention mechanism to generate a third training sequence:

[0305] The attention processing module is configured to perform processing on the second training sequence through a causal attention mechanism and a bidirectional attention mechanism to generate a fifth training sequence;

[0306] The action query module is configured to add a set of action query parameters after each token set corresponding to the image in the first positive sample in the fifth training sequence to generate a third training sequence, wherein the set of action query parameters is used to query the relationship between the plurality of token sets and the action.

[0307] In yet another optional implementation, in terms of generating the predicted action according to the third training sequence, the action generation module 1004 is specifically configured to:

[0308] perform autoregressive processing on the third training sequence based on the transformer network to generate the predicted action.

[0309] In yet another optional implementation, the predicted action includes m predicted actions, each of the m predicted actions includes n action tokens, the n action tokens correspond to actions of the predicted action in a plurality of action dimensions respectively, the set of action query parameters includes n action query parameters, and the kth action query parameter in the ith set of action query parameters in the third training sequence is used to extract relevant information of the kth action token in the n action tokens of the ith sample action in the m sample actions in the third sequence, n is a preset number of action dimensions, i is 1, 2, …, m, k is 1, 2, …, n, and m and n are positive integers.

[0310] FIG. 12 is a structural schematic diagram of a computing device according to an embodiment of the present application. As shown in FIG. 12, the computing device 120 includes a bus 1201, a processor 1202, a memory 1203, and a communication interface 1204. The processor 1202, the memory 1203, and the communication interface 1204 communicate with each other through the bus 1201. It should be understood that the present application does not limit the number of processors and memories in the computing device 120.

[0311] The bus 1201 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, only one line is shown in FIG. 12, but it does not mean that there is only one bus or only one type of bus. The bus 1201 can include a path for transmitting information between various components (e.g., the memory 1203, the processor 1202, the communication interface 1204) of the computing device 120.

[0312] The processor 1202 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), among other processors.

[0313] The memory 1203 can include volatile memory, such as random access memory (RAM), among other types of dynamic storage. The processor 1202 can also include non-volatile memory, such as read-only memory (ROM), a hard disk drive (HDD), or a solid state drive (SSD), among other types of non-volatile memory.

[0314] In an optional embodiment, the memory 1203 stores executable program code, and can also store input information, and generated target actions, among other information.

[0315] The processor 1202 executes the executable program code to implement the functions of the aforementioned action generation apparatus 100 shown in FIG. 10a or FIG. 10b, such as generating a first sequence, generating a third sequence, generating a target action, among other functions.

[0316] In yet another optional embodiment, the memory 1203 stores executable program code, and can also store a plurality of training samples, and generated predicted actions, among other information.

[0317] The memory 1203 stores executable program code, and can also store a plurality of training samples, and generated predicted actions, among other information.

[0318] The processor 1202 executes the executable program code to implement the functions of the aforementioned model training apparatus 110 shown in FIG. 11, such as generating a first training sequence, generating a third training sequence, generating a predicted action, among other functions.

[0319] The communication interface 1204 uses a transceiver module, such as but not limited to a network interface card or a transceiver, to enable communication between the computing device 120 and other devices or communication networks.

[0320] In addition, the embodiments of the present application also provide a computer readable storage medium, which stores a computer program, when the computer program is run on a processor, the method flow shown in FIG. 3 or FIG. 8 is implemented.

[0321] The embodiment of the present application further provides a computer program product, when the computer program product is run on a processor, the method flow shown in Fig. 3 or Fig. 8 is realized.

[0322] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiment described above is only schematic, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can also be electrical, mechanical or other form connection. The modules described as separate components can be or can not be physically separated, and the components displayed as modules can be or can not be physical modules, that is, can be located in one place, or can be distributed to a plurality of network modules. Part or all of the modules can be selected according to actual needs to realize the technical effects of the scheme provided in the embodiments of the present application.

[0323] In addition, each function module in the embodiments of the present application can be integrated in one processing module, or each module can be physically present alone, or two or more modules can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of software function module. When the integrated module is realized in the form of software function module and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical scheme of the present application essentially or the part of the prior art that contributes to the technical scheme, or all or part of the technical scheme can be embodied in the form of software product, and the computer software product is stored in a readable storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned readable storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and various program codes that can be stored in the medium.

[0324] In the embodiments of the present application, the words "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the embodied or designed examples are intended to present concepts in a concrete manner.

[0325] In addition, unless otherwise specified, the ordinal numbers "first", "second", etc. used in the embodiments of the present application are used to distinguish the multiple objects, and are not used to represent the order, time sequence, priority or importance of the multiple objects, for example, the first sequence and the second sequence.

[0326] The above merely illustrates the embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of the changes or replacements within the technical scope disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An action generation method, characterized in that, The method includes: A first sequence is generated based on input information, wherein the input information includes multiple types of information, including task instructions and an initial image, the task instructions being used to instruct the execution of a target task, and the initial image being used to reflect environmental information for executing the target task; The first sequence is subjected to preset processing to generate a second sequence, wherein the second sequence includes multiple token sets classified according to modality, and the multiple token sets correspond one-to-one with various information in the input information; The second sequence is processed based on the attention mechanism to generate a third sequence, wherein the third sequence includes bidirectional association information of tokens within each of the multiple token sets, and causal association information of tokens between different token sets in the multiple token sets. A target action is generated based on the third sequence, wherein the target action is an action that performs the target task.

2. The method according to claim 1, characterized in that, The various information also includes the target action and incremental image. After generating the target action based on the third sequence, the method further includes: If the target task is not completed after the target action is performed, the target action and the incremental image are added to the input information. The incremental image is used to reflect the environmental information after the target action is performed. Return to the operation of generating the first sequence based on the input information.

3. The method according to claim 2, characterized in that, The target action includes multiple action tokens, and the multiple action tokens correspond to the target action in multiple action dimensions; If the target action has a preset action token in one of the multiple action dimensions, then the target task is not completed after the target action is executed.

4. The method according to any one of claims 1-3, characterized in that, The multiple token sets are arranged according to the interaction relationships between various information in the input information, including the relationship between the task instruction and the initial image generation action.

5. The method according to claim 4, characterized in that, The step of performing preset processing on the first sequence to generate the second sequence includes: The first sequence is processed using a Markov decision process to generate a second sequence.

6. The method according to claim 5, characterized in that, The second sequence includes a set of text tokens, a set of m image tokens, and a set of m-1 action tokens. The set of text tokens corresponds to the task instructions in the input information. The i-th image token in the set of m image tokens corresponds to the i-th image in the input information. The j-th action token in the set of m-1 action tokens corresponds to the j-th action in the input information. The set of text tokens represents the target constraint in the Markov decision process. The i-th image token is the state at the i-th time step in the Markov decision process. The j-th action token is the action at the j-th time step in the Markov decision process.

7. The method according to any one of claims 4-6, characterized in that, The process of processing the second sequence based on the attention mechanism to generate the third sequence includes: The second sequence is processed using causal attention and bidirectional attention mechanisms to generate the fifth sequence; Add an action query parameter set after each token set corresponding to the image in the input information in the fifth sequence to generate a third sequence, wherein the action query parameter set is used to query the relationship between the multiple token sets and actions.

8. The method according to claim 7, characterized in that, The step of generating the target action based on the third sequence includes: The target action is generated by performing autoregressive processing on the third sequence using a transformer network.

9. The method according to claim 8, characterized in that, The target action includes n action tokens, the action query parameter set includes n action query parameters, and the kth action query parameter in the last action query parameter set in the third sequence is used to extract relevant information about the kth action token in the third sequence and the n action tokens. n is a preset number of action dimensions, and k takes the values ​​1, 2, ..., n, where n is a positive integer.

10. The method according to any one of claims 1-9, characterized in that, The method is based on an action generation model, which is used to perform the operation of generating the target action according to the input information. The action generation model is obtained from multiple training samples, including positive samples and negative samples.

11. The method according to claim 10, characterized in that, The positive samples include a first positive sample and / or a second positive sample. The first positive sample includes the task instruction, the initial image, and the reference action. The second positive sample includes the task instruction, the initial image, and an action obtained by noise processing of the reference action. The reference action is a standard action for performing the target task.

12. The method according to claim 10 or 11, characterized in that, The negative sample includes any one of the first negative sample, the second negative sample, and the third negative sample, wherein: The first negative sample includes a interference instruction, the initial image, and the reference action, wherein the interference instruction is used to instruct the execution of a task different from the target task; The second negative sample includes the task instruction, the interference image, and the reference action, wherein the interference image is used to reflect environmental information corresponding to the execution of a task different from the target task; The third negative sample includes the task instruction, the initial image, and the interference action, wherein the interference action is a standard action corresponding to the execution of a task different from the target task.

13. An action generation device, characterized in that, The device includes a serialization module, a preprocessing module, a data modeling module, and an action generation module, wherein: The serialization module is used to generate a first sequence based on input information, wherein the input information includes task instructions and an initial image, the task instructions are used to instruct the execution of a target task, and the initial image is used to reflect the environmental information corresponding to the execution of the target task; The preprocessing module is used to perform preset processing on the first sequence to generate a second sequence, wherein the second sequence includes multiple token sets classified according to modality, and the multiple token sets correspond one-to-one with various information in the input information; The data modeling module is used to process the second sequence based on the attention mechanism to generate a third sequence, wherein the third sequence includes bidirectional association information of tokens within each of the multiple token sets, and causal association information of tokens between different token sets in the multiple token sets. The action generation module is used to generate a target action based on the third sequence, wherein the target action is an action that performs the target task.

14. The apparatus according to claim 13, characterized in that, After the target action is generated based on the third sequence: The action generation module is further configured to add the target action and an incremental image to the input information if the target task is not completed after the target action, wherein the incremental image is used to reflect environmental information after the target action is performed; The action generation module is also used to trigger the serialization module to return and execute the operation of generating the first sequence based on the input information.

15. The apparatus according to claim 14, characterized in that, The target action includes multiple action tokens, and the multiple action tokens correspond to the target action in multiple action dimensions; If the target action has a preset action token in one of the multiple action dimensions, then the target task is not completed after the target action is executed.

16. The apparatus according to any one of claims 13-15, characterized in that, The multiple token sets are arranged according to the interaction relationships between various information in the input information, including the relationship between the task instruction and the initial image generation action.

17. The apparatus according to claim 16, characterized in that, In the process of performing preset processing on the first sequence to generate the second sequence, the preprocessing module is specifically used for: The first sequence is processed using a Markov decision process to generate a second sequence.

18. The apparatus according to claim 17, characterized in that, The second sequence includes a set of text tokens, a set of m image tokens, and a set of m-1 action tokens. The set of text tokens corresponds to the task instructions in the input information. The i-th image token in the set of m image tokens corresponds to the i-th image in the input information. The j-th action token in the set of m-1 action tokens corresponds to the j-th action in the input information. The set of text tokens represents the target constraint in the Markov decision process. The i-th image token is the state at the i-th time step in the Markov decision process. The j-th action token is the action at the j-th time step in the Markov decision process.

19. The apparatus according to any one of claims 16-18, characterized in that, The data modeling module includes an attention processing module and an action query module, which, in the process of processing the second sequence based on the attention mechanism to generate the third sequence: The attention processing module is used to process the second sequence through a causal attention mechanism and a bidirectional attention mechanism to generate a fifth sequence; The action query module is used to add an action query parameter set after each token set corresponding to the image in the input information in the fifth sequence to generate a third sequence, wherein the action query parameter set is used to query the relationship between the multiple token sets and actions.

20. The apparatus according to claim 19, characterized in that, In generating the target action based on the third sequence, the action generation module is specifically used for: The target action is generated by performing autoregressive processing on the third sequence using a transformer network.

21. The apparatus according to claim 20, characterized in that, The target action includes n action tokens, the action query parameter set includes n action query parameters, and the kth action query parameter in the last action query parameter set in the third sequence is used to extract relevant information about the kth action token in the third sequence and the n action tokens. n is a preset number of action dimensions, and k takes the values ​​1, 2, ..., n, where n is a positive integer.

22. An action generation device, characterized in that, The device includes a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to invoke the computer program to implement the method according to any one of claims 1-12.

23. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the method of any one of claims 1-12.

Images