Model training method and device, electronic device, storage medium, and program product

Abstract

The present disclosure provides a model training method and device, an electronic device, a storage medium and a program product. The method comprises: for multi-round model training of an action generation model, in a case where a round number of a current round is less than or equal to a target round number, training the action generation model of the current round according to sample device state data, to obtain an action generation model of a next round; in a case where the round number of the current round is greater than the target round number, simulating a device state according to first device action data generated by an action generation model of a previous round, to obtain simulated device state data of the current round, and training the action generation model of the current round according to the simulated device state data of the current round, to obtain the action generation model of the next round; and in a case where the current round is the last round, determining the action generation model of the next round as a trained action generation model. According to embodiments of the present disclosure, the performance of the action generation model can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a model training method and apparatus, electronic equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the continuous development of computer technology and artificial intelligence technology, neural network models can be applied to various task scenarios to meet practical needs. For example, in equipment control scenarios, neural network models can be used to generate motion control data for equipment in order to control the equipment to perform actions.

[0003] The neural network models of related technologies need to be trained using manually constructed training data. This training data is deterministic and rigid, and it does not take into account the action deviations and uncertainties that may occur during the execution of actions by the device. As a result, the performance of the trained neural network model is poor, and it cannot generate accurate action control data to control the device to perform actions. Therefore, how to improve the performance of neural network models has become an urgent technical problem to be solved. Summary of the Invention

[0004] This disclosure provides a model training method and apparatus, an action generation method, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] Firstly, this disclosure provides a model training method, which includes: for multi-round model training of an action generation model, when the current round number is less than or equal to the target round number, training the action generation model of the current round based on sample device state data to obtain the action generation model of the next round; when the current round number is greater than the target round number, simulating device state based on the first device action data generated by the action generation model of the previous round to obtain simulated device state data of the current round, and training the action generation model of the current round based on the simulated device state data of the current round to obtain the action generation model of the next round; when the current round is the last round, determining the action generation model of the next round as the trained action generation model, wherein the action generation model is used to generate device action data to control the target device to perform actions.

[0006] Secondly, this disclosure provides a model training device, which includes: a first model training module, a second model training module, and a model determination module.

[0007] The first model training module is used for multi-round model training of the action generation model. When the current round number is less than or equal to the target round number, the action generation model of the current round is trained based on the sample device status data to obtain the action generation model of the next round.

[0008] The second model training module is used to simulate the device state based on the first device action data generated by the action generation model of the previous round when the current round number is greater than the target round number, to obtain the simulated device state data of the current round, and to train the action generation model of the current round based on the simulated device state data of the current round to obtain the action generation model of the next round.

[0009] The model determination module is used to determine the action generation model for the next round as the trained action generation model when the current round is the last round, wherein the action generation model is used to generate device action data to control the target device to perform actions.

[0010] Thirdly, this disclosure provides an action generation method, which includes: inputting device state data of a target device into an action generation model to obtain device action data of the target device, wherein the device action data is used to control the target device to perform actions, and the action generation model is trained based on the above-described model training method.

[0011] Fourthly, this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the model training method or action generation method described above.

[0012] Fifthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described model training method or action generation method.

[0013] In a sixth aspect, this disclosure provides a computer program product that includes computer-readable code or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in a processor of an electronic device, the processor in the electronic device executes the model training method or action generation method described above.

[0014] The model training method provided in this embodiment, during multi-round model training, if the current round is less than or equal to the target round, can train the action generation model for the current round based on sample device state data, thereby ensuring that the action generation model can learn the correct action generation capability in the early training stages. Simultaneously, considering the action deviation and uncertainty of the target device during action execution, device state simulation is employed during model training. In multi-round model training, if the current round is greater than the target round, device state simulation can be performed based on the first device action data generated by the action generation model in the previous round to obtain simulated device state data for the current round, thus introducing device action deviation and uncertainty into the model training process. Then, the action generation model is trained based on the simulated device state data, enabling the trained action generation model to fully consider the device action deviation and uncertainty, generate accurate device action data, improve the performance of the action generation model, and allow the target device to accurately execute actions under the control of the device action data to meet actual needs.

[0015] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0017] Figure 1 A flowchart of a model training method provided in this embodiment of the disclosure;

[0018] Figure 2 A flowchart illustrating one model training method provided in this embodiment of the present disclosure;

[0019] Figure 3 This is a schematic diagram illustrating the application of a model training method provided in an embodiment of the present disclosure;

[0020] Figure 4 A flowchart of an action generation method provided in this embodiment of the disclosure;

[0021] Figure 5 A block diagram of a model training apparatus provided in an embodiment of this disclosure;

[0022] Figure 6This is a block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0023] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0024] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0025] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0026] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0027] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0028] The model training method according to embodiments of this disclosure can be executed by an electronic device such as a terminal device or a server. The terminal device can be an in-vehicle device, user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. The method can be implemented by a processor calling computer-readable program instructions stored in memory. Alternatively, the method can be executed by a server.

[0029] The technical terms used in the embodiments of this disclosure are explained below.

[0030] Embodied intelligent agents: These are intelligent agents with bodies that can not only "think" but also "move," and learn and complete tasks through continuous interaction with their environment via their "body." Examples include home robots, self-driving cars, and intelligent robotic arms.

[0031] Slow system: Receives language commands and visual input, and outputs a high-level task command (e.g., "go grab the cup").

[0032] Fast system (motion generator): Receives this high-level instruction and combines it with the current environmental state (such as camera view, robot joint position) to generate a series of precise motion commands (such as how many degrees the motor should rotate).

[0033] World model: This can be a technological framework used in the field of artificial intelligence to simulate environmental dynamics and predict future states.

[0034] Dynamic model: This is a technique that uses the principles of dynamics to represent a scenario as a mathematical model. This dynamic model describes the motion of a device under the influence of forces and can be applied to the precise control of complex mechanical systems such as robots and vehicles. In fields such as autonomous driving and robot control, by considering factors such as force, torque, and friction, it can more realistically reflect the dynamic behavior of the system.

[0035] Transformer is a deep learning model architecture based on the self-attention mechanism.

[0036] MLP policy network: It is a network architecture that uses a multilayer perceptron (MLP) as the policy function of the agent.

[0037] VLA model: refers to the Vision-Language-Action Model (VLA). The VLA model is an end-to-end artificial intelligence model that unifies visual perception, natural language understanding and action control into a single framework.

[0038] With the continuous development of computer technology and artificial intelligence technology, neural network models can be applied to various task scenarios to meet practical needs. For example, in equipment control scenarios, neural network models can be used to generate motion control data for equipment in order to control the equipment to perform actions.

[0039] The neural network models of related technologies need to be trained using manually constructed training data. This training data is deterministic and rigid, but it does not take into account the motion deviations and uncertainties that may occur during the execution of actions by the device. As a result, the performance of the trained neural network model is poor, and it cannot generate accurate motion control data to control the device to perform actions.

[0040] For example, in vision-language-action models based on a fast-slow dual-system architecture, high-level task planning (slow system) is typically decoupled from low-level action execution (fast system). When training such models, especially the "fast system" responsible for generating continuous action sequences (i.e., the action generator), an open-loop, one-step supervised training mode is commonly used. That is, given high-level instructions or objectives generated by the slow system, the action generator learns to predict the correct action at the current or next moment, conditioned on perfect historical states and actions from expert demonstrations.

[0041] The above-mentioned supervised training model training process has the following three prominent drawbacks.

[0042] 1. Rigid sequence length: The model is trained to make predictions within a fixed time window and lacks the inherent ability to dynamically generate or terminate action sequences according to task requirements, resulting in insufficient flexibility in actual deployment.

[0043] 2. Error Accumulation and Drift in Long-Term Execution: In real-world reasoning (such as continuous robot execution), the action generator must rely on the real-world state (which may include execution errors and environmental disturbances) generated by its previously predicted and executed actions as historical input for subsequent predictions. Since the training phase has never encountered this "imperfect history" composed of accumulated errors, the model is highly susceptible to decision drift, leading to task failure, such as the robotic arm's trajectory gradually deviating from the target or the navigation path accumulating deviations. This severely restricts the autonomy and robustness of the agent in complex, long-cycle tasks.

[0044] 3. Mismatch between training and inference scenarios: In traditional training, the action generator learns to predict the next action based on a "perfect" historical state (usually from demonstration data). However, in real inference, it faces its own imperfect historical execution results. This "exposure bias" causes the model to perform significantly worse in actual deployment than during training.

[0045] Based on this, this disclosure provides a model training method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product, as detailed in the following embodiments.

[0046] Figure 1 A flowchart illustrating a model training method provided in an embodiment of this disclosure. (Refer to...) Figure 1 The method includes steps S11 to S13.

[0047] Step S11: For multi-round model training of the action generation model, if the current round number is less than or equal to the target round number, the action generation model of the current round is trained based on the sample device status data to obtain the action generation model of the next round.

[0048] The action generation model can be understood as a model capable of generating device action information. This model can be a neural network model, a deep learning model, or a large model. In some embodiments, the action generation model can be the fast system or an action generator in a fast-slow dual-system approach. The fast system or action generator can generate corresponding single action data or a sequence of action data (i.e., device action data) for the device. In some embodiments, the action generation model can be a spatiotemporal Transformer or an MLP policy network.

[0049] In some embodiments, the device in this disclosure (such as a sample device or a target device) can perform specific actions under the control of device motion data; the device includes, but is not limited to: robots (such as sweeping robots, handling robots), industrial robotic arms, autonomous driving devices (such as autonomous vehicles, drones), etc., and this disclosure does not specifically limit the device.

[0050] Device status data can be understood as data representing the current state of a device. This device status data includes, but is not limited to, the device's position, orientation, ambient temperature, road surface smoothness, slope, etc. In some embodiments, the device status data can be visual data corresponding to the device (such as a sample device or a target device). For example, the device status data can be image data, video, or video frames from a video. In some embodiments, the visual data can be a sequence of visual observation states (such as video) used to represent the device state after the device performs an action.

[0051] The sample device status data can be actual device status data collected from the sample device; or, the sample device status data can be device status data pre-built before the start of multiple rounds of model training.

[0052] In some embodiments, based on Figure 2 It can be seen that when the current round number is less than or equal to the target round number, the action generation model for the current round is trained based on the sample device state data to obtain the action generation model for the next round, including steps S21 to S24:

[0053] Step S21: If the current round number is less than or equal to the target round number, select sample device state data for the current round from the device state sequence corresponding to the multi-round model training. The device state sequence includes multiple sample device state data generated by the sample device during the execution of actions.

[0054] Step S22: Using the motion generation model of the current wheel, generate motion based on the sample device status data of the current wheel to obtain the second device motion data of the current wheel.

[0055] Step S23: Determine the loss function based on the actual device action data corresponding to the second device action data of the current round and the sample device state data of the current round.

[0056] Step S24: Adjust the model parameters of the action generation model for the current round according to the loss function to obtain the action generation model for the next round.

[0057] Specifically, in the process of performing multi-round model training on the action generation model, if it is determined that the current round is less than or equal to the target round, in this case, sample device state data of the current round can be selected from the device state sequence corresponding to the multi-round model training. Then, the sample device state data and the task description vector are input into the action generation model of the current round to generate actions and obtain the second device action data of the current round. Among them, the task description vector is a vector obtained by performing semantic analysis on the task instructions using the task analysis model in the multi-round model training.

[0058] After obtaining the second device action data, the loss function can be calculated using the second device action data and the real device action data (i.e., sample labels) corresponding to the sample device state data of the current round. Based on the loss function, the model parameters of the action generation model of the current round are adjusted to obtain the action generation model of the next round, and the model training operation of subsequent rounds can continue.

[0059] Taking the application of the model training method provided in this disclosure in an embodied intelligence scenario as an example, the model training method is explained. In this disclosure, the device (sample device or target device) can be a robot, and the action generation model can be an action generator in a fast-slow dual system. During multi-round model training, this method needs to determine the round number of the current round (i.e., the current prediction time t of the action generation model), where T represents the round number of model training. For prediction time t (1≤t≤T), when t≤k, the historical context H_t uses the real observation sequence {o_1,...,o_t}.

[0060] Specifically, this disclosure can determine the real observation state data (i.e., sample device state data) corresponding to the current round from the real observation sequence, then fuse the real observation state data with the task condition c to obtain input data, and input the input data into the action generator for action prediction. The action generator will output the predicted action â_t at the current moment (i.e., the second device action data of the current round).

[0061] Then, this disclosure calculates the difference between the predicted action and the real expert actions recorded in the training data (i.e., the real device action data of the current round), using this difference as the loss function. Based on the calculated loss, the model parameters of the action generator are optimized and updated using the backpropagation algorithm, and the next round of model training continues, thereby achieving model optimization and iterative training until the model converges.

[0062] As can be seen from the above embodiments, the action generation model trained in this disclosure can enhance the generalization ability under complex tasks: the action generation model not only memorizes the actions of experts in specific states, but also learns a dynamic strategy on how to approach the target in the "possible state space". This enables it to still exhibit good generalization performance when facing scenes or objects that have not been directly trained on, through online adjustment.

[0063] Step S12: If the current round number is greater than the target round number, simulate the device state based on the first device action data generated by the action generation model of the previous round to obtain the simulated device state data of the current round. Then, train the action generation model of the current round based on the simulated device state data of the current round to obtain the action generation model of the next round.

[0064] The device motion data can be understood as data used to control the device (such as the sample device or the target device) in this disclosure to perform actions; the device motion data can be motion control instructions for the device, motion description text information, etc., which are not specifically limited here; for example, the device motion data can control the device to perform actions such as moving forward one meter, raising the robotic arm 10 centimeters, or taking an item from point A to point B.

[0065] It should be noted that the device motion data in this disclosure can be a single motion data or a sequence of motion data. A single motion data refers to the device motion data corresponding to any one of a series of consecutively executed motions, used to control the device to perform one of the multiple motions. For example, this single motion could be the robotic arm raising 10 centimeters. A sequence of motion data refers to the device motion data corresponding to a series of consecutively executed motions, used to control the device to continuously perform multiple motions. For example, the series of consecutively executed motions could be "identifying an item from point A, taking the item from point A, moving to point B, and placing the item at point B".

[0066] The first device motion data can be understood as the device motion data generated by the motion generation model in the previous round. It should be noted that the input data for this previous round of motion generation model can be either simulated device state data generated in the previous round or sample device state data. This previous round of motion generation model can generate motions based on the input data to obtain the first device motion data.

[0067] Simulated equipment status data can be understood as the simulated equipment status data obtained by simulating the equipment status using the first equipment action data.

[0068] In some embodiments, the first device action data is generated based on the first device state data of the previous round; the device state simulation based on the first device action data generated by the action generation model of the previous round to obtain the simulated device state data of the current round includes: performing device state simulation for a sample device using a device state simulation model based on the first device action data and the first device state data to obtain the simulated device state data of the current round; wherein, the sample device is a device used to determine the simulated device state data in multi-round model training, and the simulated device state data represents the device state of the sample device after performing an action under the control of the first device action data.

[0069] The first device state data in the previous round can be understood as the input data of the action generation model in the previous round. The action generation model in the previous round generates actions based on the input data to obtain the first device action data. The first device state data in the previous round can be the simulated device state data generated in the previous round; or, the first device state data in the previous round can be the sample device state data.

[0070] A device state simulation model can be understood as an algorithmic model used to implement device state simulation operations. This device state simulation model can be a neural network model, a simulation model, or a dynamic model.

[0071] A sample device can be understood as a device used for multi-round model training. This sample device can be used to generate training data (simulated device state data) for the action generation model during multi-round model training.

[0072] Specifically, in the process of performing a multi-round model training task on the action generation model, if it is determined that the current round is greater than the target round, then the operation of simulating the device state can be initiated. This device state simulation operation is used to generate corresponding simulated device state data for the action generation model in the current round and to train the model.

[0073] In simulating device status, this disclosure first needs to determine the input and output data (i.e., the first device status data and the first device action data) of the previous round of action generation model. Then, using the device status simulation model, device status simulation is performed on the sample device based on the first device status data and the first device action data to obtain the simulated device status data for the current round. In other words, the simulated device status data is determined based on the input and output data of the previous round of action generation model.

[0074] The device state simulation can be as follows: based on the first device state data and the first device action data, the sample device is simulated to perform an action under the control of the first device action data, and the device state data of the sample device is obtained after the action is completed.

[0075] Taking the application of the model training method provided in this disclosure in an embodied intelligence scenario as an example, the model training method is explained. In this disclosure, the device (sample device or target device) can be a robot, and the action generation model can be an action generator in a fast and slow dual system.

[0076] Based on this, the specific training scheme for multi-round model training of the action generator disclosed herein includes the following steps.

[0077] Step 1: Data preparation and system initialization.

[0078] Before training, this disclosure requires obtaining a training dataset containing task descriptions, state sequences, and corresponding action sequences. It also initializes the slow system model and the fast system model to be trained (i.e., the action generator). The slow system is used to process the task description (i.e., task instructions) and generate high-level task planning or objective conditional representations.

[0079] The state sequence can be a sequence containing sample device state data (i.e., training samples), such as a video corresponding to the robot; the action sequence can be a sequence containing multiple real device action data (i.e., training labels). The state sequence can be a training sample sequence, and the action sequence can be a training label sequence.

[0080] Step 2: Dynamically blended historical context construction.

[0081] For each training sample sequence, during multiple rounds of model training, historical context information (referred to as historical information) can be dynamically constructed for input to the action generator. This historical information is composed of two parts: real historical segments and simulated historical segments (i.e., simulated device state data).

[0082] 1. Real historical segments: Consisting of real environmental state sequences recorded in the training data from the beginning of the sequence to a certain "switching point".

[0083] The “switching point” can be a round (i.e., the target round) in a multi-round model training process; the “switching point” can be a video frame in a video.

[0084] The “switch point” is used to indicate the switching of training samples from real historical fragments to simulated historical fragments; that is, the “switch point” is used to indicate the operation of starting device state simulation.

[0085] 2. Simulated historical segment: The simulated historical segment is a simulated state generated by simulating the operation of the equipment state; from the "switching point" to the current prediction time (i.e. the current round), the simulated historical segment consists of a sequence of simulated states generated by a state transition model.

[0086] The simulated state sequence is generated as follows: the action predicted by the action generator in the previous round for the corresponding moment of the "simulated historical segment (i.e., the first device state data)" (i.e., the first device action data) is input into the state transition model (i.e., the device state simulation model), which calculates the simulated state for the next moment (i.e., the simulated device state data of the current round). It should be noted that the initial state of the simulated historical segment is the last device state data of the real historical segment.

[0087] Specifically, the execution method for constructing this dynamic hybrid history context is as follows:

[0088] 1. Determine the current round number (i.e., the current prediction time t of the action generation model).

[0089] 2. For the predicted time t (1≤t≤T), when t≤k, the historical context H_t uses the real observation sequence {o_1,...,o_t}.

[0090] 3. When t>k, the historical context H_t is composed of two parts: the first k real observations {o_1,...,o_k}, and the simulated observations {ô_{k+1},...,ô_{t-1}} from time k+1 to time t-1. Each simulated observation ô_j (j>k) is obtained recursively by inputting the state of the previous time step (i.e., the previous round) (initially o_k) and the action â_{j-1} predicted by the action generator in the previous time step into a state transition model for calculation.

[0091] The state at the previous moment can be the state data of the first device in the previous round, and the action predicted at the previous moment can be the action data of the first device in the previous round.

[0092] As can be seen from the above embodiments, the model training method provided in this disclosure is a hybrid historical condition training method for training a fast-slow dual-system decision model, aiming to improve the robustness and sequence consistency of the action generator in long-term, closed-loop task execution. The fast-slow dual system includes a slow system for high-level task planning and a fast system as an action generator, which generates low-level action sequences in an autoregressive manner. In the training phase of the action generator, this disclosure introduces a dynamic hybrid historical condition training mechanism to simulate the environmental state (i.e., simulated device state data) it faces during actual reasoning (closed-loop execution) and the cumulative influence of its past decisions. This method enables the action generator not only to learn ideal expert trajectories but also to learn how to make robust decisions under the "actual state caused by its own historical decisions," thereby gaining the ability to generate variable-length, long-term consistent action sequences.

[0093] In some embodiments, the device state simulation model includes a dynamic model or a world model; based on the first device action data and the first device state data, the device state simulation model is used to perform device state simulation for a sample device to obtain the simulated device state data for the current round, including: based on the first device action data and the first device state data, the dynamic model is used to perform device state simulation for a sample device to obtain the simulated device state data for the current round; or the first device action data and the first device state data are input into the world model, and the world model is used to perform device state simulation for a sample device to obtain the simulated device state data for the current round.

[0094] Following the previous example, during the simulation of equipment state, the simulation can be performed using a dynamic model or a world model to obtain each simulated observation ô_j (j>k). Specifically, this disclosure allows the previous state (initially o_k) and the action â_{j-1} predicted by the action generator at the previous time step to be input into a state transition model to calculate the simulated observation ô_j. This state transition model can be a pre-defined dynamic model (such as a physics simulator) or an additionally trained world model.

[0095] In other words, the implementation method and state transition process of the state transition model are important aspects of simulating environmental dynamics, and their implementation methods are flexible, including:

[0096] 1. Utilize a known dynamic model. For this dynamic model, in fields where the model is known or can be simplified (such as classical control), the precise physical equations can be directly used to construct the dynamic model; the constructed dynamic model can be a physical simulator used to simulate the actions of the sample device, thereby obtaining simulated observations ô_j (i.e., simulated device state data).

[0097] 2. Utilizing a learning-based world model. This world model can use a neural network to pre-learn or online a mapping function from "state-action to the next state," thereby simulating the actions performed by the sample device and obtaining simulated observations ô_j (i.e., simulated device state data). It should be noted that this world model can be trained jointly with the action generator or separately.

[0098] As can be seen from the above embodiments, this disclosure provides multiple methods such as dynamic models and world models for simulating equipment states, enabling this disclosure to flexibly select appropriate methods for simulating equipment states according to actual conditions, thereby improving the applicability of this disclosure.

[0099] In some embodiments, the device state simulation is performed based on the first device action data generated by the action generation model in the previous round to obtain the simulated device state data for the current round, including: using the first device action data to control the sample device to perform an action; when the sample device completes the action, the device state data of the sample device is collected, and the device state data is determined as the simulated device state data for the current round.

[0100] It should be noted that collecting the device status data of the sample device includes: collecting the device status data of the sample device through the corresponding sensor or data acquisition device; or collecting the device status data of the sample device through the corresponding visual data acquisition device (such as a camera).

[0101] Continuing with the previous example, during the simulation of the device state, each simulated observation o_j (j>k) can be obtained using a real sample device. Specifically: In the training of the physical robot (i.e., the sample device), this disclosure uses the predicted action â_{j-1} from the previous moment to control the physical robot to perform actions. After the action is completed, sensors or cameras are used directly to read the actual next-moment observation o_j of the physical robot, and o_j is used as the simulated observation data. At this point, although the state is real, the causal chain it generates originates from the model's own predicted actions, rather than expert actions.

[0102] In other words, the device state simulation operation performed in this disclosure can use feedback from a real physical system. During training on the physical robot platform, â_{j-1} is sent to the robot for execution, and then new environmental observations o_j are directly read as the "simulated state" through cameras, force sensors, etc. This is the most realistic "state transition," perfectly reflecting the decision-making closed loop.

[0103] As can be seen from the above embodiments, this disclosure provides multiple methods, such as real physical system feedback, for simulating device status, enabling this disclosure to flexibly select appropriate device status simulation methods according to actual conditions, thereby improving the applicability of this disclosure.

[0104] In some embodiments, the motion generation model for the current round is trained based on the simulated device state data of the current round to obtain the motion generation model for the next round, including steps one and two.

[0105] Step 1: Using the motion generation model of the current wheel, generate motions based on the simulated equipment status data to obtain the second equipment motion data of the current wheel.

[0106] The second device motion data can be understood as the motion generation model of the current wheel, which generates device motion data based on the simulated device state data.

[0107] In some embodiments, the action generation model of the current round is used to generate actions based on the simulated device state data to obtain the second device action data of the current round. This includes: inputting the simulated device state data and the task description vector corresponding to the multi-round model training into the action generation model of the current round to generate actions and obtain the second device action data of the current round. The task description vector is a vector obtained by performing semantic analysis on the task instructions using the task analysis model in the multi-round model training.

[0108] This task analysis model can be understood as a model that performs semantic analysis on task instructions. For example, the task analysis model can be a large model, a deep learning model, etc. In some embodiments, the task analysis model can be the slower system in a fast-slow dual-system setup. Based on this, the task description vector can be a task planning condition vector c output by the slower system, which encodes the target semantics of the task instructions.

[0109] The task instruction can be a command to control the device to perform a specific task. This instruction can be text-based, voice-based, or similar. For example, when a user needs to control a robot to perform a specific action via voice, the user's voice instruction (i.e., the task instruction) is input into a task analysis model for analysis, obtaining a task description vector. The action generation model then generates device action data based on this task description vector, thereby controlling the robot to perform the corresponding action.

[0110] Specifically, this disclosure allows the task instructions to be input into the task analysis model during multi-round model training for semantic analysis to obtain a task description vector. After obtaining the simulated device state data, the simulated device state data and the task description vector can be combined (e.g., concatenated, fused, etc.), and the combined simulated device state data and task description vector can be input into the action generation model of the current round to generate actions and obtain the second device action data of the current round.

[0111] Following the previous example, this disclosure iterates the training process of the motion generator in multiple batches. For each training sequence in a batch (e.g., a record of a robot completing a "grasp-place" task, including the visual observation state sequence o_1, o_2, ..., o_T and the robotic arm joint motion sequence a_1, a_2, ..., a_T, and the task instruction "place the red block into the blue box"), the system of this disclosure performs the following operations:

[0112] 1. Advanced Condition Generation: Utilize a slow system (such as a large language model) to process text-based task instructions and output a task planning condition vector c, which encodes the target semantics of the task.

[0113] 2. In the process of using the action generator to generate actions (i.e., action prediction), this disclosure can fuse the constructed historical context H_t (such as historical information constructed based on the above-mentioned equipment state simulation operation) with the task condition c (i.e., task description vector) to obtain input data, and input the input data into the action generator for action prediction. The action generator will output the predicted action â_t at the current moment (i.e. the second equipment action data of the current round).

[0114] As can be seen from the above embodiments, this disclosure can utilize slow systems for advanced task planning and fuse the generated semantic condition vectors with historical context to guide the action generator in making predictions; thereby significantly improving the success rate and logical coherence of robots when facing multi-stage, long-sequence tasks, and reducing action errors caused by a lack of global planning.

[0115] Step 2: Based on the motion data of the second device and the actual motion data of the current round, train the motion generation model of the current round to obtain the motion generation model of the next round.

[0116] The real device action data can be understood as sample labels during the model training process.

[0117] Based on this, it can be seen that during multi-round model training, the action generation model of the current round can be used to perform action generation operations based on the simulated device state data to obtain the second device action data for the current round. Then, based on the second device action data and the real device action data of the current round, the model parameters of the action generation model of the current round are adjusted to obtain the action generation model for the next round. In this way, the device's action deviation and uncertainty are introduced into the model training process, enabling the trained action generation model to fully consider the device's action deviation and uncertainty, generate accurate device action data, and improve the performance of the action generation model.

[0118] In some embodiments, training the action generation model for the current round based on the second device action data and the real device action data of the current round to obtain the action generation model for the next round includes: selecting real device action data for the current round from the device action sequences corresponding to the multi-round model training, wherein the device action sequence includes multiple real device action data, and the multiple real device action data are used to control the sample device to perform actions; determining a loss function based on the second device action data and the real device action data of the current round; and adjusting the model parameters of the action generation model for the current round based on the loss function to obtain the action generation model for the next round.

[0119] Following the previous example, this disclosure allows for the execution of conditional action prediction and loss calculation during multi-round model training. Specifically, this method combines a dynamic hybrid historical context constructed from simulated device state operations with high-level task planning conditions generated by a slow system, and inputs both into the action generator. The action generator then predicts the action at the current moment (i.e., the second device action data for the current round) based on these conditions.

[0120] Then, this disclosure calculates the difference between the predicted action and the real expert action recorded in the training data (i.e., the real device action data of the current round) as a loss function. Alternatively, this disclosure uses the difference between the predicted action at the prediction time of an entire sequence or a part of a sequence (including the second device action data of the current round and the device action data generated in historical rounds) and the real expert action recorded in the training data of the same sequence length as a loss function.

[0121] Then, based on the calculated loss, the model parameters of the action generator are optimized and updated using the backpropagation algorithm. The next round of model training then continues, thus achieving model optimization and iterative training until the model converges.

[0122] It should be noted that, in the process of loss calculation and gradient backpropagation, this disclosure calculates the difference (such as mean squared error) between the predicted action â_t and the real expert action a_t, and then calculates the gradient. In the "hybrid mode", the gradient will be backtracked to the prediction of the action generator at an earlier time by simulating the observation sequence and the state transition model, thereby achieving the training objective of allowing the model to learn the influence of its early decisions on the long-term state and the final action selection.

[0123] As can be seen from the above embodiments, the model training method provided by this disclosure can endow the action generation model with the ability to generate variable-length action sequences. By allowing the action generation model to become accustomed to making continuous decisions based on its "self-generated history" during training, the model inherently possesses the ability to dynamically adjust decisions according to real-time states and objectives until the task is completed, breaking through the limitation of fixed window length and significantly improving deployment flexibility. Furthermore, this disclosure can fundamentally improve the robustness and consistency of long-term execution: the action generation model is exposed to simulated state sequences caused by its own predicted actions during the training phase, thereby learning to self-correct and robustly plan under "real" conditions with execution errors, sensor noise, and model inaccuracies. This greatly reduces the risk of execution drift, failure, or loss of control due to error accumulation in long-cycle tasks.

[0124] Step S13: If the current round is the last round, determine the action generation model for the next round as the trained action generation model, wherein the action generation model is used to generate device action data to control the target device to perform actions.

[0125] In this context, the target device can be understood as the device that needs to be controlled by the device action data generated by the trained action generation model in a real-world application scenario.

[0126] It should be noted that the sample device and the target device in this embodiment can be the same device or two identical devices. When the sample device and the target device are two identical devices, these two devices (i.e., the sample device and the target device) can be devices with the same configuration parameters or hardware attributes. For example, the sample device and the target device can be two robotic arms of the same model or two robots of the same model.

[0127] When the sample device and the target device are the same device, the sample device can be used to generate training data for the action generation model. After completing multiple rounds of training operations for the action generation model, the trained action generation model can be used to generate device action data to control the target device (i.e., the sample device) to perform actions.

[0128] When the sample device and the target device are the same two devices, since the sample device and the target device have the same configuration parameters or hardware attributes, after using the sample device to generate training data for the action generation model and completing multiple rounds of training operations for the action generation model, the device action data generated by the trained action generation model can also effectively control the target device to perform actions, thereby achieving accurate control of the target device.

[0129] In some embodiments, the method may also train the action generation model using only the simulated device state data obtained from device state simulation. Specifically, the model training method of this disclosure further includes: when the target round is zero, for the first round of training in a multi-round model training, using the action generation model trained in the first round, generating actions based on preset device state data to obtain the third device action data of the first round of model training; training the action generation model trained in the first round based on the third device action data and the real device action data of the first round of model training to obtain the action generation model of the next round; then, when the round number of the current round is greater than the target round, simulating the device state based on the first device action data generated by the action generation model of the previous round to obtain the simulated device state data of the current round, and training the action generation model of the current round based on the simulated device state data of the current round to obtain the action generation model of the next round; when the current round is the last round, determining the action generation model of the next round as the trained action generation model, wherein the action generation model is used to generate device action data to control the target device to perform actions.

[0130] Among them, the third device motion data can be understood as the device motion data output by the motion generation model in the first round of model training.

[0131] Predicted device state data can be understood as the initial state data of the sample device. With zero target rounds, device state simulation is required for each round of model training. Since the device state simulation process requires the device action data generated by the action generation model in the previous round, but the first round of model training does not have the action generation model from the previous round, preset device state data is provided for the action generation model in the first round of model training.

[0132] Specifically, when the target number of rounds is zero, for the first round of multi-round model training, preset device state data can be input into the action generation model of the first round of model training to generate actions, obtaining the third device action data of the first round of model training. A loss function is calculated based on this third device action data and the real device action data of the first round of model training, and the model parameters of the action generation model are adjusted based on this loss function to obtain the action generation model for the next round. Then, the next round of model training continues. Since the target number of rounds is zero, the number of rounds in the next round of model training is greater than the target number of rounds, thus triggering device state simulation based on the first device action data generated by the action generation model of the previous round, obtaining the simulated device state data for the current round, and training the action generation model for the current round based on the simulated device state data for the current round to obtain the action generation model for the next round, until the current round is the last round.

[0133] As can be seen from the above embodiments, this disclosure fully considers the problem that manually constructed training data is deterministic and rigid training data. Therefore, in the process of multi-round model training, the simulated device state data obtained by device state simulation is used to train the action generation model, thereby ensuring that the trained action generation model has better performance. Furthermore, this disclosure can fully consider various factors that may lead to a decrease in model performance during the training process and provides corresponding solutions, thereby improving the applicability of this disclosure.

[0134] In some embodiments, the model training process of the action generation model includes multiple model training batches, each model training batch performing multiple rounds of model training; the method further includes: generating a state simulation start value for the current model training batch, wherein the state simulation start value is used to indicate whether the current model training batch starts device state simulation, and the state simulation start value is positively correlated with the batch number of the current model training batch; when the state simulation start value indicates that the current model training batch starts device state simulation, determining a target round according to the device state sequence corresponding to the current model training batch, wherein the target round is the round in which device state simulation is started in the multiple rounds of model training that controls the current model training batch, and the target round is a round randomly selected from the multiple rounds of model training, or a round selected from the multiple rounds of model training and negatively correlated with the batch number of the current model training batch.

[0135] The state simulation initiation value can be understood as a value used to indicate whether the current model training batch should initiate device state simulation in multiple model training batches. For example, the state simulation initiation value can be a probability value or a model prediction confidence level.

[0136] Following the previous example, the model training process of this disclosure is performed iteratively in batches. For each training sequence in a batch, it is necessary to determine whether to initiate device state simulation. If it is determined that device state simulation is to be initiated, this disclosure performs the step of dynamically constructing historical context. Specifically, this disclosure first determines a probability (i.e., the state simulation initiation value) based on a preset strategy (e.g., with probabilities of 0.5 or 0.6), and decides whether the current training sequence enters "mixed mode (i.e., initiating device state simulation)" based on this probability.

[0137] Alternatively, this disclosure also provides a progressive planned sampling approach where, in the early stages of training, the probability of enabling "mixed mode" is set to 0, utilizing only expert data to allow the model to quickly learn the basic policy. As training progresses, this probability is increased linearly or exponentially to an upper limit (e.g., 0.8), allowing the model to gradually adapt to processing its own decision history.

[0138] The specific details of this approach are as follows: In the early stages of training (such as the first three batches of multiple model training sessions), the probability of enabling "hybrid mode" is set to 0, meaning that only expert data (i.e., sample device status data) is used, without using simulated observation sequences (i.e., simulated device status data). The purpose of this is to allow the model to quickly learn the basic strategy and understand the general task situation.

[0139] In the later stages of training (such as all batches after the first three batches in multiple model training batches), this probability is increased linearly or exponentially until it reaches an upper limit (such as 0.8). That is, as the number of iterations increases, the probability continuously increases, and the chance of triggering the "hybrid mode" is higher (i.e., the state simulation initiation value is positively correlated with the number of batches in the current model training batch). The purpose of this is to enable the trained agent to adapt to the uncertainty of the real physical world more quickly and safely.

[0140] Based on the above, according to this probability, if it is determined not to enter (i.e., not to activate "hybrid mode"), then the actual observation sequence will be used as the training data throughout. If it enters (i.e., activates "hybrid mode"), then the target round will be determined based on the device state sequence corresponding to the current model training batch.

[0141] Alternatively, the strategy for determining the "switching point" disclosed herein is a technical aspect of constructing a dynamic historical context, which includes, but is not limited to, a planned sampling strategy. This planned sampling strategy can generate a probability value (i.e., a state simulation start value) based on a probability function that monotonically increases with the number of training epochs, and determine whether to initiate "hybrid mode" based on this probability value.

[0142] As can be seen from the above embodiments, this disclosure can flexibly determine whether to enable device state simulation in multiple model training batches according to the actual situation, thereby reducing the consumption of computing resources caused by multiple device state simulations and avoiding the waste of computing resources.

[0143] In some embodiments, the device state sequence includes a device state video, which includes a target device state video frame determined from multiple device state video frames, and the model training round corresponding to the target device state video frame is determined as the target round. The multiple device state video frames correspond one-to-one with the multiple rounds of model training in the current model training batch. The target device state video frame is a video frame randomly selected from the multiple device state video frames, or a video frame selected from the multiple device state video frames that is negatively correlated with the batch number of the current model training batch.

[0144] Following the previous example, when it is determined that "hybrid mode" will be activated, this disclosure can randomly select a switching point k upon activation. This switching point k can be understood as the frame number corresponding to a video frame in the device status video; for example, the switching point k could be the frame number corresponding to a video frame at one-third of the total length of the video sequence. This switching point k is used to control when to activate the device status simulation operation in the current batch of multi-round model training, where 0 ≤ k ≤ T. It should be noted that the number of samples in the sample sequence (such as video) of this disclosure corresponds one-to-one with the number of rounds of training.

[0145] In this context, "equipment status video" can be understood as a video recording or demonstrating the operational status of the sample equipment, such as a video recording the actions of a robotic arm during operation. An equipment status video frame can be a video frame within an existing equipment status video.

[0146] In some embodiments, this disclosure may employ a random strategy to select the target round, regardless of whether a state simulation start value is used for judgment. This random strategy may randomly select a video frame (i.e., the target state video frame) within the effective time range of the training sequence (i.e., the video). Since the number of video frames corresponds to the number of rounds in the multi-round model training, and one video frame corresponds to one model training round, the model training round corresponding to this video frame can be determined as the target round. When the current round in the multi-round model training is greater than the target round, the device state simulation operation is initiated. That is, this disclosure determines the switching point k by randomly selecting the switching point k, including: randomly selecting a video frame within the effective time range of the sample sequence (i.e., the video). Since the number of video frames corresponds to the number of rounds in the multi-round model training, and one video frame corresponds to one model training round, the model training round corresponding to this video frame can be determined as the target round. When the current round in the multi-round model training is greater than the target round, the device state simulation operation is initiated. For example, a sample sequence may contain 100 samples (such as video frames). Multi-round model training requires 100 rounds, with each round using one sample for model training. The two are interrelated. For instance, the switching point K refers to the Kth action video frame in the action video sequence. During multi-round model training, the round corresponding to the Kth action frame can be used to determine whether to switch the real observation sequence to a simulated observation sequence for model training.

[0147] In some embodiments, for multiple model training batches, this disclosure provides other switching strategies for the "hybrid mode," which can be implemented in various ways to optimize training results. This switching strategy can be a course-learning switching approach, where, in the early stages of training, the switching point k is set near the end of the sequence, primarily training the model to handle short-term self-errors. As the model's capabilities improve, k is gradually moved towards the beginning of the sequence, increasing the length of the simulation history and enhancing the model's ability to handle long-term error accumulation. Specifically, the course-learning switching is implemented as follows: In the early stages of training (e.g., the first three batches in multiple model training batches), the switching point k can be set at the end of the sequence, thereby reducing the proportion of the "simulated observation sequence" in the training samples. As the model's capabilities improve, in the later stages of training (e.g., all batches after the first three batches in multiple model training batches), k can be gradually moved towards the beginning of the sequence, thereby increasing the proportion of the "simulated observation sequence" in the training samples (i.e., the target device status video frames selected from multiple device status video frames and negatively correlated with the batch number of the current model training batch).

[0148] As can be seen from the above embodiments, this disclosure takes into account the complexity of actual training scenarios and can provide different training strategies to determine the target device state video frames, thereby achieving the purpose of adapting to complex actual training scenarios. This allows the model training operation for the action generation model to be successfully executed in different scenarios to meet actual needs.

[0149] In some embodiments, the timing of the “switching point” in this disclosure may be a time point randomly selected in the training sequence; during the process of triggering the “hybrid mode” based on a probability, the probability may be randomly selected under the control of a probability function that changes with the training process, and the probability may be a selection triggered based on the model prediction confidence.

[0150] Through the above implementation method, the action generator not only learns "what actions should be taken in an ideal state" during training, but more importantly, it learns "how to correct and continue to complete the task now, given the decisions it made previously that may have caused the state to deviate from the ideal trajectory." This training mechanism enables it to autonomously, coherently, and stably generate action sequences of arbitrary length based on actual perceived states when deployed independently, until the task is completed, significantly improving the practicality of the agent in complex, long-cycle tasks.

[0151] The model training method provided in this disclosure takes into account the action deviations and uncertainties of the target device during the execution of actions, and adopts device state simulation during the model training process. In the current round of multi-round model training, device state simulation can be performed based on the first device action data generated by the action generation model in the previous round to obtain the simulated device state data for the current round, thereby introducing the action deviations and uncertainties of the device into the model training process. Then, action generation is performed based on the simulated device state data to obtain the second device action data for the current round, and training is performed based on the second device action data and the real device action data. This allows the trained action generation model to fully consider the action deviations and uncertainties of the device, generate accurate device action data, improve the performance of the action generation model, and enable the target device to accurately execute actions under the control of the device action data to meet actual needs.

[0152] Figure 3 This is a schematic diagram illustrating the application of a model training method provided in an embodiment of this disclosure, based on... Figure 3 As can be seen, the motion generation model disclosed herein can run in an electronic device and interact with a robotic arm (i.e., the target device); the motion generation model can generate motions based on the device status data of the robotic arm, obtain device motion data for the robotic arm, and send the device motion data to the robotic arm; the robotic arm will execute the motions based on the device motion data, thereby completing the corresponding work task or operation.

[0153] The action generation model can be an action generator in a fast-slow system, which also includes a slow system. Before applying this action generator in a real-world scenario, it needs to be trained. Based on this, this disclosure provides a training system to improve the consistency of long-range action sequences of an embodied agent. This training system includes the following modules: a training data loading module, a slow system processing module, a dynamic historical context construction module, an action generator (fast system) module, a loss calculation module, and a model optimization module. The training process of this system is iterative in batches, and the specific training method is shown in the following steps.

[0154] Step 1: Data preparation and system initialization.

[0155] The training data loading module is used to obtain a training dataset containing a task description, a state sequence, and corresponding action sequences. The state sequence can be a visual observation state sequence {o_1,o_2,...,o_T} (i.e., video). The action sequence can be a robotic arm joint action sequence {a_1,a_2,...,a_T}. The task instruction includes "Place the red block into the blue box".

[0156] Step 2: Advanced conditions generation.

[0157] The task instructions are processed using a slow system (such as a large language model) in the slow system processing module, and a task planning condition vector c is output; this vector encodes the target semantics of the task.

[0158] Step 3: Dynamically construct the historical context.

[0159] The dynamic historical context building module first determines whether the current training sequence enters "mixed mode" based on a preset strategy (e.g., a probability value of 0.5 or model prediction confidence). If it does not enter "mixed mode," the entire sequence uses the real observation sequence as the history. If it does enter, the following steps are performed:

[0160] 1. Randomly select a switching point k (e.g., one-third of the total length of the state sequence).

[0161] 2. During the multi-round model training process, determine the current prediction time t (the round number of the current round).

[0162] 3. When t≤k, the historical context H_t is the sequence of actual observed states {o_1,...,o_t}.

[0163] 4. When t>k, the historical context H_t is composed of two parts: the first k real observations {o_1,...,o_k}, and the simulated observations from time k+1 to time t-1 {ô_{k+1},...,ô_{t-1}}.

[0164] Each simulated observation ô_j (j>k) is obtained recursively by inputting the state of the previous time step (initially o_k) and the action â_{j-1} predicted by the action generator in the previous time step into a state transition model for calculation. This state transition model can be a pre-set physical simulator, an additionally trained world model, or, in the training of the physical robot, directly using the actual next-time observation o_j read by the sensor after executing â_{j-1}.

[0165] Step 4: Action prediction.

[0166] After fusing the constructed historical context H_t with the task condition c, it is input into the action generator (such as a spatiotemporal Transformer or MLP policy network) module, and the predicted action â_t at the current time is output.

[0167] Step 5: Loss calculation and gradient backpropagation.

[0168] Using the loss calculation module, calculate the difference (such as mean squared error) between the predicted action â_t and the real expert action a_t, and calculate the gradient.

[0169] Using the model optimization module, the model parameters of the action generator are optimized and updated based on the calculated loss through the backpropagation algorithm.

[0170] Repeat the above multiple rounds of model training to iteratively train the model until it converges.

[0171] Based on the above steps, this disclosure provides a method for training an action generator in a fast-slow dual-system decision model. During the training process, a dynamic historical context is constructed for the action generator. Within a single training sequence, the historical context changes from being entirely composed of expert demonstration data in the early stages to being composed of a combination of some expert data and some simulated state data in the later stages. The simulated state data originates from the result of inputting the action predicted by the action generator in the preceding time step into a state transition process.

[0172] The historical context dynamic construction module can, according to the switching strategy, feed back the predicted action of the action generator to a state transition model after a specific point in the training sequence, and use the state output by the model to concatenate with the previous real state as the input condition for the action generator in subsequent time steps.

[0173] The model training method described above achieves a smooth transition from open-loop training to closed-loop execution: this disclosure effectively simulates the dynamics of closed-loop execution during training, narrowing the "simulation-reality" gap between training and deployment. This enables trained agents to adapt to the uncertainties of the real physical world more quickly and safely.

[0174] This disclosure enhances the generalization ability of action generators for complex tasks: the action generator not only memorizes the actions of experts in specific states, but also learns a dynamic strategy for approaching the target in the "possible state space". This enables it to exhibit good generalization performance even when faced with scenes or objects that have not been directly trained on, through online tuning.

[0175] This disclosure achieves modularity and versatility: it is a training framework-level design that does not depend on a specific network structure and can be widely applied to various VLA models, robot policy learning, and autonomous driving decision-making models that adopt fast and slow system architectures to improve the long-term quality of their sequential decisions.

[0176] In summary, this disclosure aims to provide an improved training method to address the aforementioned problems of action generators in fast-slow dual-system VLA models. The purpose of this disclosure is to enable the action generator to learn how to handle real-world states resulting from its past decisions during the training phase through a hybrid historical conditional training mechanism. This endows the generator with the ability to generate variable-length action sequences and significantly improves its robustness, temporal consistency, and task success rate in long-term, closed-loop execution.

[0177] Figure 4 A flowchart illustrating an action generation method provided in an embodiment of this disclosure. (Refer to...) Figure 4 The method includes step S41: inputting the device status data of the target device into the action generation model to obtain the device action data of the target device, wherein the device action data is used to control the target device to perform actions, and the action generation model is trained based on the above model training method.

[0178] For an explanation of the action generation model and target device in this embodiment, please refer to the content of the model training method described above, which will not be repeated here.

[0179] For example, this disclosure provides an embodied intelligent agent system, including a slow system for task understanding and planning, a fast system for action generation, and a corresponding target device (such as a robot). The fast system (i.e., the action generator) is trained using the model training method described above. The action generator can receive a state sequence derived from its own historical prediction results as input conditions, and generate a continuous, consistent, and task-oriented action sequence based on this, and has the generalization ability to generate variable-length action sequences.

[0180] The action generation method provided in this disclosure takes into account the action deviation and uncertainty of the target device during the action execution process. It uses the action generation model trained by the above-mentioned model training method to generate device action data of the target device, so that the target device can accurately execute actions under the control of the device action data to meet actual needs.

[0181] The above is an illustrative scheme of an action generation method according to this embodiment. It should be noted that the technical solution of this action generation method belongs to the same concept as the technical solution of the model training method described above. For details not described in detail in the technical solution of the action generation method, please refer to the description of the technical solution of the model training method described above.

[0182] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0183] In addition, this disclosure also provides a model training device, an electronic device, and a computer-readable storage medium, all of which can be used to implement any of the model training methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the relevant section on methods and will not be repeated here.

[0184] Figure 5 This is a block diagram of a model training device provided in an embodiment of the present disclosure.

[0185] Reference Figure 5 This disclosure provides a model training device, which includes: a first model training module 501, a second model training module 502, and a model determination module 503.

[0186] The first model training module 501 is used for multi-round model training of the action generation model. When the current round number is less than or equal to the target round number, the action generation model of the current round is trained based on the sample device status data to obtain the action generation model of the next round.

[0187] The second model training module 502 is used to simulate the device state based on the first device action data generated by the action generation model of the previous round when the current round number is greater than the target round number, to obtain the simulated device state data of the current round, and to train the action generation model of the current round based on the simulated device state data of the current round to obtain the action generation model of the next round.

[0188] The model determination module 503 is used to determine the action generation model for the next round as the trained action generation model when the current round is the last round. The action generation model is used to generate device action data to control the target device to perform actions.

[0189] In some embodiments, the second model training module 502 is further configured to: generate actions based on simulated device state data using the action generation model of the current round to obtain the second device action data of the current round; and train the action generation model of the current round based on the second device action data and the real device action data of the current round to obtain the action generation model of the next round.

[0190] In some embodiments, the first device action data is generated based on the first device state data of the previous round; the second model training module 502 is further configured to: perform device state simulation for the sample device using the device state simulation model according to the first device action data and the first device state data, and obtain the simulated device state data of the current round; wherein, the sample device is the device used to determine the simulated device state data in the multi-round model training, and the simulated device state data represents the device state of the sample device after performing an action under the control of the first device action data.

[0191] In some embodiments, the device state simulation model includes a dynamic model or a world model; the second model training module 502 is further configured to: perform device state simulation for the sample device using the dynamic model based on the first device action data and the first device state data to obtain the simulated device state data for the current round; or input the first device action data and the first device state data into the world model, and perform device state simulation for the sample device using the world model to obtain the simulated device state data for the current round.

[0192] In some embodiments, the second model training module 502 is further configured to: control the sample device to perform actions using the first device action data; and, when the sample device completes the action, collect the device status data of the sample device and determine the device status data as the simulated device status data for the current round.

[0193] In some embodiments, the model training process of the action generation model includes multiple model training batches, each model training batch performing multiple rounds of model training; the model training device further includes a simulation startup module, configured to: generate a state simulation startup value for the current model training batch, wherein the state simulation startup value is used to indicate whether the current model training batch starts device state simulation, and the state simulation startup value is positively correlated with the batch number of the current model training batch; when the state simulation startup value indicates that the current model training batch starts device state simulation, determine a target round according to the device state sequence corresponding to the current model training batch, wherein the target round is the round in the multiple rounds of model training that controls the current model training batch to start device state simulation, and the target round is a round randomly selected from the multiple rounds of model training, or a round selected from the multiple rounds of model training that is negatively correlated with the batch number of the current model training batch.

[0194] In some embodiments, the device state sequence includes a device state video, which includes multiple device state video frames. These multiple device state video frames are video frames collected by the sample device during the execution of an action. The simulation startup module is further configured to: determine a target device state video frame from the multiple device state video frames, and determine the model training round corresponding to the target device state video frame as the target round. The multiple device state video frames correspond one-to-one with the multiple rounds of model training in the current model training batch. The target device state video frame is a video frame randomly selected from the multiple device state video frames, or a video frame selected from the multiple device state video frames that is negatively correlated with the batch number of the current model training batch.

[0195] In some embodiments, the second model training module 502 is further configured to: input the simulated device state data and the task description vector corresponding to the multi-round model training into the action generation model of the current round to generate actions and obtain the second device action data of the current round, wherein the task description vector is a vector obtained by performing semantic analysis on the task instructions using the task analysis model in the multi-round model training.

[0196] In some embodiments, the second model training module 502 is further configured to: select real device action data for the current round from the device action sequence corresponding to the multi-round model training, wherein the device action sequence includes multiple real device action data, and the multiple real device action data are used to control the sample device to perform actions; determine a loss function based on the second device action data and the real device action data for the current round; and adjust the model parameters of the action generation model for the current round based on the loss function to obtain the action generation model for the next round.

[0197] In some embodiments, the first model training module 501 is further configured to: when the current round number is less than or equal to the target round number, select sample device state data for the current round from the device state sequence corresponding to the multi-round model training, wherein the device state sequence includes multiple sample device state data generated by the sample device during the execution of an action; use the action generation model of the current round to generate actions based on the sample device state data of the current round to obtain the second device action data of the current round; determine a loss function based on the second device action data of the current round and the real device action data corresponding to the sample device state data of the current round; and adjust the model parameters of the action generation model of the current round according to the loss function to obtain the action generation model for the next round.

[0198] The model training device provided in this embodiment, during multi-round model training, if the current round is less than or equal to the target round, can train the action generation model for the current round based on sample device state data, thereby ensuring that the action generation model can learn the correct action generation capability in the early training stages. Simultaneously, considering the action deviation and uncertainty of the target device during action execution, device state simulation is employed during model training. In multi-round model training, if the current round is greater than the target round, device state simulation can be performed based on the first device action data generated by the action generation model in the previous round to obtain simulated device state data for the current round, thus introducing device action deviation and uncertainty into the model training process. Then, the action generation model is trained based on the simulated device state data, enabling the trained action generation model to fully consider the device action deviation and uncertainty, generate accurate device action data, improve the performance of the action generation model, and allow the target device to accurately execute actions under the control of the device action data to meet actual needs.

[0199] The above is an illustrative scheme of a model training device according to this embodiment. It should be noted that the technical solution of this model training device and the technical solution of the model training method described above belong to the same concept. For details not described in detail in the technical solution of the model training device, please refer to the description of the technical solution of the model training method described above.

[0200] Figure 6 This is a block diagram of an electronic device provided in an embodiment of the present disclosure.

[0201] Reference Figure 6This disclosure provides an electronic device, which includes: at least one processor 601; at least one memory 602; and one or more I / O interfaces 603 connected between the processor 601 and the memory 602; wherein the memory 602 stores one or more computer programs that can be executed by the at least one processor 601, and the one or more computer programs are executed by the at least one processor 601 to enable the at least one processor 601 to perform the above-described model training method or action generation method.

[0202] This disclosure also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the aforementioned model training method or action generation method. The computer-readable storage medium may be volatile or non-volatile.

[0203] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described model training method or action generation method.

[0204] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0205] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0206] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0207] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0208] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0209] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0210] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0211] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0212] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0213] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A model training method, characterized in that, include: For multi-round model training of the action generation model, if the current round number is less than or equal to the target round number, the action generation model of the current round is trained based on the sample device status data to obtain the action generation model of the next round. If the current round is greater than the target round, the device state is simulated based on the first device action data generated by the action generation model of the previous round to obtain the simulated device state data of the current round. Based on the simulated device state data of the current round, the action generation model of the current round is trained to obtain the action generation model of the next round. If the current round is the last round, the action generation model for the next round is determined as the trained action generation model, wherein the action generation model is used to generate device action data to control the target device to perform actions.

2. The method according to claim 1, characterized in that, The step of training the motion generation model for the current round based on the simulated device state data of the current round to obtain the motion generation model for the next round includes: Using the motion generation model of the current wheel, motion is generated based on the simulated device state data to obtain the second device motion data of the current wheel; Based on the second device motion data and the actual device motion data of the current round, the motion generation model of the current round is trained to obtain the motion generation model of the next round.

3. The method according to claim 1, characterized in that, The first device action data is generated based on the first device status data from the previous round; The step of simulating the device state based on the first device action data generated by the action generation model in the previous round to obtain the simulated device state data for the current round includes: Based on the first device action data and the first device status data, the device status simulation model is used to perform device status simulation for the sample device to obtain the simulated device status data for the current round. The sample device is used to determine the simulated device state data during the multi-round model training. The simulated device state data represents the device state of the sample device after performing an action under the control of the first device action data.

4. The method according to claim 3, characterized in that, The equipment state simulation model includes a dynamic model or a world model; The step of performing equipment state simulation for the sample equipment using an equipment state simulation model based on the first equipment action data and the first equipment state data to obtain the simulated equipment state data for the current round includes: Based on the first device action data and the first device state data, the dynamic model is used to perform a device state simulation for the sample device to obtain the simulated device state data for the current wheel; or The first device action data and the first device state data are input into the world model, and the world model is used to perform device state simulation for the sample device to obtain the simulated device state data for the current round.

5. The method according to claim 1, characterized in that, The step of simulating the device state based on the first device action data generated by the action generation model in the previous round to obtain the simulated device state data for the current round includes: Using the motion data of the first device, control the sample device to perform actions; When the sample device completes its action, the device status data of the sample device is collected, and the device status data is determined as the simulated device status data for the current round.

6. The method according to any one of claims 1 to 5, characterized in that, The model training process of the action generation model includes multiple model training batches, and each model training batch performs multiple rounds of model training. The method further includes: Generate a state simulation start value for the current model training batch, wherein the state simulation start value is used to indicate whether the current model training batch starts device state simulation, and the state simulation start value is positively correlated with the batch number of the current model training batch; When the state simulation start value indicates that the current model training batch starts device state simulation, the target round is determined according to the device state sequence corresponding to the current model training batch. The target round is the round in which device state simulation is started in the multi-round model training that controls the current model training batch. The target round is either a round randomly selected from the multi-round model training or a round selected from the multi-round model training that is negatively correlated with the number of batches of the current model training batch.

7. The method according to claim 6, characterized in that, The device status sequence includes a device status video, which includes multiple device status video frames. These multiple device status video frames are video frames captured by the sample device during the execution of an action. Determining the target round based on the device state sequence corresponding to the current model training batch includes: From the plurality of device status video frames, a target device status video frame is determined, and the model training round corresponding to the target device status video frame is determined as the target round. The plurality of device status video frames correspond one-to-one with the multiple rounds of model training in the current model training batch. The target device status video frame is a video frame randomly selected from the plurality of device status video frames, or a video frame selected from the plurality of device status video frames that is negatively correlated with the batch number of the current model training batch.

8. The method according to any one of claims 2 to 5, characterized in that, The step of using the motion generation model of the current wheel to generate motion based on the simulated device state data to obtain the second device motion data of the current wheel includes: The simulated device state data and the task description vector corresponding to the multi-round model training are input into the action generation model of the current round to generate actions, thereby obtaining the second device action data of the current round. The task description vector is a vector obtained by performing semantic analysis on task instructions using the task analysis model in the multi-round model training.

9. The method according to claim 2, characterized in that, The step of training the motion generation model for the current round based on the second device motion data and the actual device motion data for the current round to obtain the motion generation model for the next round includes: From the device action sequences corresponding to multiple rounds of model training, select the real device action data of the current round, wherein the device action sequence includes multiple real device action data, and the multiple real device action data are used to control the sample device to perform actions; The loss function is determined based on the second device motion data and the actual device motion data of the current round; The model parameters of the action generation model for the current round are adjusted based on the loss function to obtain the action generation model for the next round.

10. The method according to any one of claims 1 to 5, characterized in that, When the current round number is less than or equal to the target round number, the action generation model for the current round is trained based on sample device status data to obtain the action generation model for the next round, including: If the current round number is less than or equal to the target round number, sample device state data for the current round is selected from the device state sequence corresponding to the multi-round model training. The device state sequence includes multiple sample device state data generated by the sample device during the execution of actions. Using the motion generation model of the current round, motion is generated based on the sample device state data of the current round to obtain the second device motion data of the current round; Based on the second device action data of the current round and the real device action data corresponding to the sample device state data of the current round, the loss function is determined; The model parameters of the action generation model for the current round are adjusted based on the loss function to obtain the action generation model for the next round.

11. A model training device, characterized in that, include: The first model training module is used for multi-round model training of the action generation model. When the current round number is less than or equal to the target round number, the action generation model of the current round is trained based on the sample device status data to obtain the action generation model of the next round. The second model training module is used to simulate the device state based on the first device action data generated by the action generation model of the previous round when the current round number is greater than the target round number, to obtain the simulated device state data of the current round, and to train the action generation model of the current round based on the simulated device state data of the current round to obtain the action generation model of the next round. The model determination module is used to determine the action generation model for the next round as the trained action generation model when the current round is the last round, wherein the action generation model is used to generate device action data to control the target device to perform actions.

12. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-10.

14. A computer program product, characterized in that, Includes computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the method as described in any one of claims 1-10.

Images