Visual language action model training method and device based on reinforcement learning, electronic equipment and storage medium

Abstract

The invention relates to a visual language action model training method and device based on reinforcement learning, electronic equipment and a storage medium, and the method comprises the steps: initializing an original visual language action model, and obtaining an initial visual language action model which has the capability of generating action elements; controlling the mechanical arms in different environments to perform action and environment state interaction with the initial visual language action model to obtain action tracks corresponding to the mechanical arms; updating an initial visual language action model based on the plurality of action tracks; and iteratively executing the interaction and the updating operation until a first stop condition is met, thereby obtaining the visual language action model. According to the method, the visual language action performance is still remarkably improved under the condition of data scarcity, and the dependence on large-scale manual annotation data is greatly reduced. And new operation modes which do not appear in supervised learning can emerge. And the accuracy of processing long-time-domain tasks and double-arm exercise tasks by the visual language action model is improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method and apparatus for training visual language action models based on reinforcement learning, as well as electronic devices and storage media. Background Technology

[0002] In recent years, Vision-Language-Action (VLA) models have become an important research paradigm in the field of robotics manipulation. These models unify visual perception, language understanding, and action generation within a single framework, enabling robots to understand complex multimodal commands and perform diverse physical manipulation tasks. The emergence of VLA models has provided new possibilities for solving the problem of flexible robot manipulation in real-world environments, demonstrating broad application prospects in tasks such as grasping, assembly, and navigation.

[0003] To train high-performance VLA models, existing research typically employs a two-stage training strategy. The first stage is large-scale pre-training, which trains the model on rich multimodal data, including human operation videos, image-text pairs, and heterogeneous robot datasets, enabling the model to acquire basic vision-language-action alignment capabilities. The second stage is supervised fine-tuning (SFT), which enhances the model's performance on specific tasks by fine-tuning on high-quality robot trajectory data. This paradigm has achieved significant progress, particularly in improving the generalization ability of VLA models through large-scale pre-training.

[0004] However, with the expansion of model size and the increasing performance requirements for downstream tasks, the existing SFT training paradigm faces two key challenges. The first challenge is data scarcity. Expanding SFT requires a large amount of high-quality robot trajectory data operated by humans. Collecting this type of data requires not only carefully designed experimental scenarios and diverse manipulators, but also skilled operators, resulting in extremely high data acquisition costs and limited scale. Unlike natural language processing, where massive amounts of text data can be easily obtained, the difficulty and cost of collecting robot trajectory data make it difficult to effectively scale the data, severely restricting the scalability of VLA models. The second challenge is insufficient generalization ability. Because SFT relies on limited, scenario- and task-specific data, when VLA models encounter unseen tasks, environments, or objects, especially in composite tasks, long-term tasks, or real-world scenarios with distributional shifts, model performance often drops significantly. This limited generalization ability becomes a key bottleneck restricting the practical deployment of VLA models. Summary of the Invention

[0005] In view of this, this disclosure proposes a training scheme for a visual language action model.

[0006] According to one aspect of this disclosure, a method for training a visual language action model is provided. The method includes: initializing an original visual language action model to obtain an initial visual language action model, wherein the initial visual language action model has the ability to generate action elements; controlling a robotic arm in different environments to interact with the initial visual language action model in terms of actions and environmental states to obtain action trajectories corresponding to each robotic arm, wherein the action trajectory includes action elements corresponding to multiple actions performed by the robotic arm for a task and an environmental image; updating the initial visual language action model based on the multiple action trajectories; and iteratively executing the interaction and update operations until a first stopping condition is met to obtain a visual language action model.

[0007] In one possible implementation, controlling the robotic arms in different environments to interact with the initial visual language action model to obtain the motion trajectory corresponding to each robotic arm includes: inputting the input data corresponding to the first time step into the initial visual language action model to obtain the action element corresponding to the second time step; the input data includes: environmental image, task description, and action element; based on the action element corresponding to the second time step, controlling the robotic arms to perform actions and obtaining the input data corresponding to the second time step; using the second time step as a new first time step, iteratively executing the input until the second time step is used as a new first time step, until a second stopping condition is met to obtain the motion trajectory.

[0008] In one possible implementation, updating the initial visual language action model based on multiple action trajectories includes: obtaining a task status identifier, the task status identifier indicating whether the task corresponding to the action trajectory has been successfully executed; if the task status identifiers of the action trajectories corresponding to the same task are the same, then the action trajectories corresponding to the same task are taken as action trajectories to be removed; removing the action trajectories to be removed, and taking the remaining action trajectories as high-quality action trajectories; and updating the initial visual language action model based on the high-quality action trajectories.

[0009] In one possible implementation, updating the initial visual language action model based on the high-quality motion trajectory includes: determining the importance sampling probability of each action element executable by the robotic arm based on the high-quality motion trajectory; determining the reward value corresponding to the high-quality motion trajectory using a binary allocation mechanism based on the task status identifier; and inputting a policy update objective function based on the importance sampling probability and the reward value to update the parameters of the initial visual language action model, wherein the policy update objective function does not include a relative entropy divergence regularization term.

[0010] In one possible implementation, determining the importance sampling probability corresponding to each action element that the robotic arm can execute based on the high-quality motion trajectory includes: determining the initial importance sampling probability corresponding to each action element that the robotic arm can execute based on the environmental image contained in the high-quality motion trajectory; and cropping each of the initial importance sampling probabilities based on a first threshold range to obtain the importance sampling probability, wherein the first threshold range is an asymmetric threshold, and the span of the upper limit distance 0 of the first threshold range is greater than the span of the lower limit distance 0.

[0011] In one possible implementation, obtaining the task status identifier includes: if the environment is a virtual environment, receiving the task status identifier returned by the virtual environment after performing an action in the virtual environment; if the environment is a real environment, processing an environmental image containing the corresponding real environment based on a large language model to obtain the task status identifier.

[0012] In one possible implementation, controlling the robotic arm to interact with the initial visual language action model in different environments includes: control based on an environment process pool, and executing the interaction operations in parallel for the different environments.

[0013] According to another aspect of this disclosure, a visual language action model training device is provided. The device includes:

[0014] An initial visual language action model acquisition unit is used to initialize the original visual language action model to obtain an initial visual language action model, wherein the initial visual language action model has the ability to generate action elements.

[0015] The motion trajectory acquisition unit is used to control the robotic arms in different environments to interact with the initial visual language motion model to obtain the motion trajectory corresponding to each robotic arm. The motion trajectory includes the motion elements corresponding to the multiple actions performed by the robotic arms and the environmental image.

[0016] The model update unit is used to update the initial visual language action model based on multiple action trajectories;

[0017] The visual language action model determination unit is used to iteratively execute the interaction and the update operation until the first stopping condition is met, thereby obtaining the visual language action model.

[0018] In one possible implementation, the motion trajectory acquisition unit is further configured to:

[0019] The input data corresponding to the first time step is input into the initial visual language action model to obtain the action element corresponding to the second time step; the input data includes: environmental image, task description, and action element.

[0020] Based on the action elements corresponding to the second time step, control the robotic arm to perform actions and obtain the input data corresponding to the second time step;

[0021] The second time step is used as the new first time step, and the operation of using the input to use the second time step as the new first time step is executed iteratively until the second stopping condition is met, thus obtaining the motion trajectory.

[0022] In one possible implementation, the model update unit is further configured to:

[0023] Obtain the task status identifier, which indicates whether the task corresponding to the action trajectory has been executed successfully;

[0024] If the task status identifiers of the action trajectories corresponding to the same task are the same, the action trajectories corresponding to the same task will be regarded as action trajectories to be removed.

[0025] Remove the motion trajectories to be removed and use the remaining motion trajectories as high-quality motion trajectories;

[0026] The initial visual language action model is updated based on the high-quality motion trajectory.

[0027] In one possible implementation, the model update unit is further configured to:

[0028] Based on the high-quality motion trajectory, the importance sampling probability of each motion element that the robotic arm can execute is determined.

[0029] Based on the task status identifier, the reward value corresponding to the high-quality action trajectory is determined using a binary allocation mechanism.

[0030] Based on the importance sampling probability and the reward value, the policy update objective function is input to update the parameters of the initial visual language action model. The policy update objective function does not include a relative entropy divergence regularization term.

[0031] In one possible implementation, the model update unit is further configured to:

[0032] Based on the environmental image contained in the high-quality motion trajectory, the initial importance sampling probability corresponding to each of the action elements that the robotic arm can execute is determined;

[0033] The importance sampling probability is obtained by pruning each of the initial importance sampling probabilities based on the first threshold range. The first threshold range is an asymmetric threshold, and the span of the upper limit distance 0 of the first threshold range is greater than the span of the lower limit distance 0.

[0034] In one possible implementation, the model update unit is further configured to:

[0035] When the environment is a virtual environment, after performing an action in the virtual environment, the task status identifier returned by the virtual environment is received;

[0036] When the environment is a real environment, the task status identifier is obtained by processing an environmental image that reflects the real environment based on a large language model.

[0037] In one possible implementation, the action trajectory acquisition unit is further used to: execute the interactive operation in parallel for different environments based on the control of the environment process pool.

[0038] According to another aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.

[0039] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0040] According to another aspect of this disclosure, a computer program product is provided, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0041] In this embodiment, a reinforcement learning-based online interactive learning mechanism is used to sample the robotic arm's motion trajectory, thereby training the initial visual-language-action model and ultimately training the robotic arm's operation strategy. This method significantly improves visual-language-action performance even under data-scarce conditions, greatly reducing reliance on large-scale manually labeled data. Furthermore, it can generate new operation modes not found in supervised learning. This enhances the accuracy of the visual-language-action model in handling long-term tasks and dual-arm learning tasks, providing an efficient, scalable, and easily deployable technical solution to the problems of data scarcity and insufficient generalization in robot operation tasks.

[0042] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0043] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0044] Figure 1 This is a flowchart illustrating the visual language action model training method provided in an embodiment of this disclosure.

[0045] Figure 2 A schematic diagram of the structure of the visual language action model training device provided in the embodiments of this disclosure.

[0046] Figure 3 This is a schematic diagram of the structure of an electronic device for training a visual language action model, provided in an embodiment of this disclosure. Detailed Implementation

[0047] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0048] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0049] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0050] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0051] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0052] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0053] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions.

[0054] Figure 1 This is a flowchart illustrating the visual language action model training method provided in an embodiment of this disclosure. Figure 1 As shown, the method includes:

[0055] S11, Initialize the original visual language action model to obtain an initial visual language action model, which has the ability to generate action elements.

[0056] An initial visual language model can generate motion elements, but the accuracy of these motion elements is too low to accurately complete the task. Motion elements can represent the degrees of freedom of one or more joints of a robotic arm. For example, motion elements can be generated using temperature sampling (utilizing temperature parameters).

[0057] In one example, a robotic arm can be controlled to complete a first task. This embodiment of the disclosure does not limit the type of the first task. The first task is a task that the robotic arm can complete after training. Action elements corresponding to each movement during the completion of the first task, as well as environmental images corresponding to each movement, are collected. The action elements, environmental images, and task description of the first task are used as demonstration data. The task description can be a text description. The demonstration data is manually annotated data. For example, one or several demonstration data sets can be used for each type of first task. Thus, using a small amount of demonstration data, the original visual language action model can be initialized, resulting in an initial visual language action model.

[0058] S12, control the robotic arms in different environments to interact with the initial visual language action model to obtain the action trajectory corresponding to each robotic arm. The action trajectory includes the action elements corresponding to the multiple actions performed by the robotic arms and the environmental image.

[0059] The environment can include factors such as the color, quantity, shape, size, and task type of objects other than the robotic arm in the scene where it performs the task. The interaction here can be the robotic arm performing actions according to the output data of the initial visual language model, and the initial visual language model outputting new data based on the state of the environment after the actions are performed. The output data can include action elements. The environment image can record the state of the environment. The actions performed by the robotic arm can correspond to action elements and environment images. The action elements and environment images corresponding to the actions obtained during task execution can be used as action trajectories.

[0060] For any given environment, the number of interactions determines the number of actions the robotic arm performs. A single environment can correspond to at least one motion trajectory. Thus, one or more motion trajectories can be obtained for different environments.

[0061] S13, based on the multiple action trajectories, update the initial visual language action model.

[0062] Multiple motion trajectories can correspond to one or more environments and / or one or more tasks. Therefore, updating the parameters of the initial visual language action model using multiple motion trajectories can not only improve model accuracy but also enhance its generalization ability. This helps improve the adaptability of the visual language action model to new environments and / or new tasks after deployment, and also improves the matching degree between the output action elements and the task.

[0063] S14, iteratively execute the interaction and the update operation until the first stopping condition is met to obtain the visual language action model.

[0064] The first stopping condition indicates model convergence. For example, the difference in model parameters after two consecutive iterations is less than a difference threshold, or the number of iterations reaches a threshold. It should be noted that in each iteration, S14 follows S13; that is, each iteration involves multiple interactions before updating the parameters of the initial visual-language action model. The initial visual-language action model that satisfies the first stopping condition is taken as the final visual-language action model.

[0065] In this embodiment, a reinforcement learning-based online interactive learning mechanism is used to sample the robotic arm's motion trajectory, thereby training the initial visual-language-action model and ultimately training the robotic arm's operation strategy. This method significantly improves visual-language-action performance even under data-scarce conditions, greatly reducing reliance on large-scale manually labeled data. Furthermore, it can generate new operation modes not found in supervised learning. This enhances the accuracy of the visual-language-action model in handling long-term tasks and dual-arm learning tasks, providing an efficient, scalable, and easily deployable technical solution to the problems of data scarcity and insufficient generalization in robot operation tasks.

[0066] In one possible implementation, controlling the robotic arms in different environments to interact with the initial visual language action model to obtain the motion trajectory corresponding to each robotic arm includes: inputting the input data corresponding to the first time step into the initial visual language action model to obtain the action element corresponding to the second time step; the input data includes: environmental image, task description, and action element; based on the action element corresponding to the second time step, controlling the robotic arms to perform actions and obtaining the input data corresponding to the second time step; using the second time step as a new first time step, iteratively executing the input until the second time step is used as a new first time step, until a second stopping condition is met to obtain the motion trajectory.

[0067] The first time step is earlier than the second time step. The initial visual-language action model processes the input data to obtain action elements. For example, the initial visual-language action model can generate action elements through temperature sampling. The number of action elements is not limited here. Furthermore, the time step corresponding to each action element is later than the time step corresponding to the input data. The task description in the input data for each time step corresponding to the same environment can be the same. In this way, as the time steps increase, the initial visual-language model updates the action elements to guide the robotic arm to perform actions that match the task and environment as much as possible.

[0068] The environmental images in the input data can indicate the state of the environment at the first time step for the initial visual language action model, making the action elements output by the initial visual language action model at the second time step more closely aligned with the environment and task. The task description in the input data can indicate the final goal of the action elements output by the initial visual language model at the second time step. The action elements in the input data can provide the initial visual language action model with a basic reference for nearby time steps, allowing for the adjustment of new action elements based on the action elements in the input data.

[0069] The second stopping condition can be that the accumulated time steps reach a preset time step threshold, or the second stopping condition can be a task status indicator indicating that the task has succeeded. The task status indicator can be obtained based on the environment.

[0070] The interaction methods and content provided in this embodiment can make the motion trajectory fit the task and environment as closely as possible, thereby improving the usability of the motion trajectory.

[0071] In one possible implementation, updating the initial visual language action model based on multiple action trajectories includes: obtaining a task status identifier, the task status identifier indicating whether the task corresponding to the action trajectory has been successfully executed; if the task status identifiers of the action trajectories corresponding to the same task are the same, then the action trajectories corresponding to the same task are taken as action trajectories to be removed; removing the action trajectories to be removed, and taking the remaining action trajectories as high-quality action trajectories; and updating the initial visual language action model based on the high-quality action trajectories.

[0072] The task status flag can contain two values, indicating task success and task failure respectively. Task success means the task was completed and the expected result was achieved. Task failure means the task was not completed, or the maximum time step has been reached, but the expected result was not achieved.

[0073] In this embodiment, the same task can be a single task within the same environment. Multiple action trajectories can be obtained for the same task. These multiple action trajectories may not be completely identical or may be entirely different. If the task state identifier corresponding to the same task is the same, the action trajectory corresponding to that same task is removed, and the remaining action trajectory is taken as the high-quality action trajectory. Updating the parameters of the initial visual language action model using high-quality action trajectories can reduce the probability of the advantage estimate being 0, and can make the gradient of the updated parameters more stable. It can also evenly propagate the reward at the action trajectory level to each action element, thereby improving the training effectiveness of the initial visual language action model.

[0074] In one possible implementation, updating the initial visual language action model based on the high-quality motion trajectory includes: determining the importance sampling probability of each action element executable by the robotic arm based on the high-quality motion trajectory; determining the reward value corresponding to the high-quality motion trajectory using a binary allocation mechanism based on the task status identifier; and inputting a policy update objective function based on the importance sampling probability and the reward value to update the parameters of the initial visual language action model, wherein the policy update objective function does not include a relative entropy divergence regularization term.

[0075] In this embodiment, the importance sampling probability of preset action elements executable by the robotic arm can be determined based on high-quality motion trajectories. For example, the action elements executable by the robotic arm could be the degrees of freedom of the first joint of the index finger or the degrees of freedom of the wrist joint of the robotic arm. The above are merely examples of action elements and do not exhaust all possibilities of action elements. The importance sampling probability can characterize the first difference between the execution probability of the action element under the new strategy and the execution probability of the same action element under the old strategy. The first difference can guide the initial visual language action model to output action elements that are more in line with the task and environment under the new strategy. The time step corresponding to the new strategy can be the next time step after the time step corresponding to the old strategy.

[0076] The binary allocation mechanism allows setting only two candidate reward values ​​for each high-quality motion trajectory: a first reward value and a second reward value. If the task status indicates success, the high-quality motion trajectory receives the first reward value. If the task status indicates failure, the high-quality motion trajectory receives the second reward value. In one example, the first reward value can be 1, and the second reward value can be 0.

[0077] In this embodiment, reinforcement learning algorithms and policy update objective functions within those algorithms can be used. Group Relative Policy Optimization (GRPO) is preferred. The parameters of the initial visual-language action model are updated based on the reward values ​​corresponding to high-quality action trajectories and the sampling probabilities of each importance level. Furthermore, the policy update objective function does not include a relative entropy divergence regularization term, thus reducing memory consumption during parameter updates without compromising training accuracy.

[0078] In this embodiment, the reward value is a trajectory-level reward, not a reward for a single action. This simplifies the reward mechanism (reward function design) and significantly improves the scalability and cross-environment transferability of the visual language action model. It also enhances the visual language action model's ability to handle long-term tasks. Furthermore, this embodiment specifically removes the relative entropy divergence regularization term, allowing the entire update process to balance model training effectiveness with resource conservation. The saved resources can then be used for more update processes, further improving training efficiency.

[0079] In one possible implementation, determining the importance sampling probability of each action element that the robotic arm can execute based on the high-quality motion trajectory includes: determining the initial importance sampling probability of each action element that the robotic arm can execute based on the environmental image contained in the high-quality motion trajectory; and cropping each of the initial importance sampling probabilities based on a first threshold range to obtain the importance sampling probability, wherein the first threshold range is an asymmetric threshold range, and the span of the upper limit distance 0 of the first threshold range is greater than the span of the lower limit distance 0.

[0080] Environmental images can represent environmental states, and the initial visual-language action model can determine the initial importance sampling probability based on the environmental state provided by the environmental image. A first threshold range can be preset and used to crop the initial importance sampling probability. This first threshold range can be an asymmetric threshold range, meaning that the center position of the first threshold range does not correspond to 0, but contains 0. Furthermore, the upper limit of the first threshold range is greater than the lower limit of the first threshold range from 0.

[0081] After cropping, the importance sampling probability is obtained. Using the importance sampling probability in this embodiment to update the parameters of the initial visual language action model can promote the exploration of strategies and improve the adaptability of the visual language action model to new environments and tasks, as well as the accuracy of task execution.

[0082] In one possible implementation, obtaining the task status identifier includes: when the environment is a virtual environment, obtaining the task status identifier in response to receiving information returned by the virtual environment that represents the task completion status; and when the environment is a real environment, obtaining the task status identifier in response to receiving environmental image processing results provided by a large language model.

[0083] In this embodiment, the environment can be a virtual environment or a real environment. In a virtual environment, the robotic arm is a virtual robotic arm. After the virtual robotic arm performs an action, the virtual environment can send information reflecting the task completion status. This information may include a task status identifier. This allows the task status identifier to be obtained.

[0084] In a real-world environment, the robotic arm is a real robotic arm. After the robotic arm performs an action, the image acquisition device can send an environmental image reflecting the real environment to a large language model. The large language model can process the environmental image to obtain information representing the task completion status. This information can include a task status identifier. The image acquisition device can be set up in the real environment. For example, it can be set up on the robotic arm.

[0085] In this embodiment, the environment is not limited to a real or virtual environment, which expands the diversity of environments and increases the diversity of acquired motion trajectories. Furthermore, task status identifiers can be acquired for different environments, improving the feasibility of determining high-quality motion trajectories under various conditions.

[0086] In one possible implementation, controlling the robotic arm to interact with the initial visual language action model in different environments includes: control based on an environment process pool, and executing the interaction operations in parallel for the different environments.

[0087] In this embodiment, an environment process pool is set up. The environment process pool can manage and control various environments in parallel, so as to control the robotic arm to interact with the initial visual language action model and the environment state in different environments. This significantly improves the efficiency of motion trajectory sampling and reduces computational costs.

[0088] Figure 2 This is a schematic diagram of the structure of the visual language action model training device provided in an embodiment of this disclosure. Figure 2 As shown, the device 20 includes:

[0089] The initial visual language action model acquisition unit 21 is used to initialize the original visual language action model to obtain the initial visual language action model, which has the ability to generate action elements.

[0090] The motion trajectory acquisition unit 22 is used to control the robotic arm in different environments to interact with the initial visual language motion model to obtain the motion trajectory corresponding to each robotic arm. The motion trajectory includes the motion elements corresponding to the multiple actions performed by the robotic arm and the environmental image.

[0091] The model update unit 23 is used to update the initial visual language action model based on multiple action trajectories;

[0092] The visual language action model determination unit 24 is used to iteratively execute the interaction and the update operation until the first stopping condition is met to obtain the visual language action model.

[0093] In one possible implementation, the motion trajectory acquisition unit 22 is further configured to:

[0094] The input data corresponding to the first time step is input into the initial visual language action model to obtain the action element corresponding to the second time step; the input data includes: environmental image, task description, and action element.

[0095] Based on the action elements corresponding to the second time step, control the robotic arm to perform actions and obtain the input data corresponding to the second time step;

[0096] The second time step is used as the new first time step, and the operation of using the input to use the second time step as the new first time step is executed iteratively until the second stopping condition is met, thus obtaining the motion trajectory.

[0097] In one possible implementation, the model update unit 23 is further configured to:

[0098] Obtain the task status identifier, which indicates whether the task corresponding to the action trajectory has been executed successfully;

[0099] If the task status identifiers of the action trajectories corresponding to the same task are the same, the action trajectories corresponding to the same task will be regarded as action trajectories to be removed.

[0100] Remove the motion trajectories to be removed and use the remaining motion trajectories as high-quality motion trajectories;

[0101] The initial visual language action model is updated based on the high-quality motion trajectory.

[0102] In one possible implementation, the model update unit 23 is further configured to:

[0103] Based on the high-quality motion trajectory, the importance sampling probability of each motion element that the robotic arm can execute is determined.

[0104] Based on the task status identifier, the reward value corresponding to the high-quality action trajectory is determined using a binary allocation mechanism.

[0105] Based on the importance sampling probability and the reward value, the policy update objective function is input to update the parameters of the initial visual language action model. The policy update objective function does not include a relative entropy divergence regularization term.

[0106] In one possible implementation, the model update unit 23 is further configured to:

[0107] Based on the environmental image contained in the high-quality motion trajectory, the initial importance sampling probability corresponding to each of the action elements that the robotic arm can execute is determined;

[0108] The importance sampling probability is obtained by pruning each of the initial importance sampling probabilities based on the first threshold range. The first threshold range is an asymmetric threshold, and the span of the upper limit distance 0 of the first threshold range is greater than the span of the lower limit distance 0.

[0109] In one possible implementation, the model update unit 23 is further configured to:

[0110] When the environment is a virtual environment, after performing an action in the virtual environment, the task status identifier returned by the virtual environment is received;

[0111] When the environment is a real environment, the task status identifier is obtained by processing an environmental image that reflects the real environment based on a large language model.

[0112] In one possible implementation, the motion trajectory acquisition unit 22 is further used to: execute the interactive operation in parallel for different environments based on the control of the environment process pool.

[0113] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0114] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0115] This disclosure also provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0116] This disclosure also provides a computer program product, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above method.

[0117] Figure 3 This is a schematic diagram of the structure of an electronic device for training a visual language action model according to an embodiment of this disclosure. For example, device 1900 can be provided as a server or terminal device. (Refer to...) Figure 3 The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0118] Device 1900 may also include a power supply component 1926 configured to perform power management of device 1900, a wired or wireless network interface 1950 configured to connect device 1900 to a network, and an input / output interface 1958 (I / O interface). Device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM macOS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0119] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of the device 1900 to perform the above-described method.

[0120] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0121] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0122] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions to implement various aspects of this disclosure.

[0123] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0124] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0125] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0126] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0127] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for training a visual language action model, characterized in that, include: Initialize the original visual language action model to obtain an initial visual language action model, which has the ability to generate action elements; Control the robotic arms in different environments to interact with the initial visual language action model to obtain the action trajectory corresponding to each robotic arm. The action trajectory includes the action elements corresponding to the multiple actions performed by the robotic arm and the environmental image. The initial visual language action model is updated based on multiple action trajectories. The interaction and update operation are executed iteratively until the first stopping condition is met, thus obtaining the visual language action model.

2. The method according to claim 1, characterized in that, The process of controlling the robotic arms in different environments to interact with the initial visual language action model to obtain the motion trajectory corresponding to each robotic arm includes: The input data corresponding to the first time step is input into the initial visual language action model to obtain the action element corresponding to the second time step; the input data includes: environmental image, task description, and action element. Based on the action elements corresponding to the second time step, control the robotic arm to perform actions and obtain the input data corresponding to the second time step; The second time step is used as the new first time step, and the operation of using the input to use the second time step as the new first time step is executed iteratively until the second stopping condition is met, thus obtaining the motion trajectory.

3. The method according to claim 1, characterized in that, The step of updating the initial visual language action model based on multiple action trajectories includes: Obtain the task status identifier, which indicates whether the task corresponding to the action trajectory has been executed successfully; If the task status identifiers of the action trajectories corresponding to the same task are the same, the action trajectories corresponding to the same task will be regarded as action trajectories to be removed. Remove the motion trajectories to be removed and use the remaining motion trajectories as high-quality motion trajectories; The initial visual language action model is updated based on the high-quality motion trajectory.

4. The method according to claim 3, characterized in that, The step of updating the initial visual language action model based on the high-quality motion trajectory includes: Based on the high-quality motion trajectory, the importance sampling probability of each motion element that the robotic arm can execute is determined. Based on the task status identifier, the reward value corresponding to the high-quality action trajectory is determined using a binary allocation mechanism. Based on the importance sampling probability and the reward value, the policy update objective function is input to update the parameters of the initial visual language action model. The policy update objective function does not include a relative entropy divergence regularization term.

5. The method according to claim 4, characterized in that, The determination of the importance sampling probability of each action element executable by the robotic arm based on the high-quality motion trajectory includes: Based on the environmental image contained in the high-quality motion trajectory, the initial importance sampling probability corresponding to each of the action elements that the robotic arm can execute is determined; The importance sampling probability is obtained by pruning each of the initial importance sampling probabilities based on the first threshold range. The first threshold range is an asymmetric threshold, and the span of the upper limit distance 0 of the first threshold range is greater than the span of the lower limit distance 0.

6. The method according to claim 3, characterized in that, The acquisition of the task status identifier includes: When the environment is a virtual environment, after performing an action in the virtual environment, the task status identifier returned by the virtual environment is received; When the environment is a real environment, the task status identifier is obtained by processing an environmental image that reflects the real environment based on a large language model.

7. The method according to claim 1, characterized in that, Controlling the robotic arm to interact with the initial visual language action model in different environments, including: Based on the control of the environment process pool, the interactive operations are executed in parallel for different environments.

8. A visual language action model training device, characterized in that, include: An initial visual language action model acquisition unit is used to initialize the original visual language action model to obtain an initial visual language action model, wherein the initial visual language action model has the ability to generate action elements. The motion trajectory acquisition unit is used to control the robotic arms in different environments to interact with the initial visual language motion model to obtain the motion trajectory corresponding to each robotic arm. The motion trajectory includes the motion elements corresponding to the multiple actions performed by the robotic arms and the environmental image. The model update unit is used to update the initial visual language action model based on multiple action trajectories; The visual language action model determination unit is used to iteratively execute the interaction and the update operation until the first stopping condition is met, thereby obtaining the visual language action model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images