Model training method for action sequence generation, electronic device, and storage medium

Abstract

This application provides a model training method, electronic device, and storage medium for action sequence generation. Relating to the field of artificial intelligence, the method includes: removing target actions from an initial action sequence to obtain a training sequence; generating multiple action sequences with different execution orders using a target model based on the training sequence; and iteratively training the target model according to the action differences among the multiple action sequences to obtain a trained target model. In the technical solution of this application, by removing some actions, the ability to automatically complete missing actions bidirectionally is achieved. This application solves the technical problem in related technologies where models cannot fully utilize the potential inverse correlation between actions, thus making it difficult for models to achieve bidirectional action completion capabilities.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a model training method, electronic device, and storage medium for action sequence generation. Background Technology

[0002] Intelligent agents are driving the transformation of traditional automation towards intelligence. Through autonomous decision-making and continuous learning capabilities, intelligent agents can understand and handle complex business processes like human employees, autonomously completing tasks from data extraction to system operation. For example, when receiving an email instruction to "process an urgent purchase order," the intelligent agent can automatically log into the system, identify and fill out the purchase form, handle abnormal pop-ups, and finally provide feedback on the results to relevant personnel.

[0003] Currently, in order to control intelligent agents to perform tasks, reinforcement learning is used to improve data utilization efficiency and enhance generalization ability. These technologies are generally divided into two main categories: model-free and model-based. Model-free methods rely on multiple interactions with the real environment, resulting in high cost and low efficiency. Model-based methods can utilize environmental simulation to generate virtual data and improve training efficiency, but are prone to cumulative performance degradation or training instability due to model inference errors. Furthermore, the methods provided in these technologies involve unidirectional trajectory generation, which is not conducive to optimizing the overall strategy from different state distributions, leading to low algorithm sample utilization and difficulty in achieving reliable convergence of the model in complex and variable scenarios, thus hindering the improvement of intelligent agent model performance. Summary of the Invention

[0004] This application provides a model training method, electronic device, and storage medium for action sequence generation, in order to alleviate or solve the technical problem in related technologies where models cannot fully utilize the potential inverse correlation between actions, thus making it difficult for models to achieve bidirectional action completion capabilities.

[0005] In a first aspect, embodiments of this application provide a model training method for action sequence generation, including: Remove the target action from the initial action sequence to obtain the training sequence; Based on the training sequence, the target model is used to generate multiple action sequences with different execution orders; Based on the differences in actions across multiple action sequences, the target model is iteratively trained to obtain a well-trained target model.

[0006] Secondly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods of embodiments of this application when executing the computer program.

[0007] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of any one of the embodiments of this application.

[0008] Fourthly, embodiments of this application provide a computer program product, including a computer program, which, when executed by a processor, implements any of the methods described in the embodiments of this application.

[0009] Based on the model training method for action sequence generation described in the first aspect above, this application has at least the following beneficial effects or advantages: By removing key target actions and actively disrupting the integrity of the initial action sequence, the model is forced to learn to generate diverse and reasonable action sequences from incomplete inputs, thereby improving the model's generalization ability and robustness in complex scenarios. By using multiple action execution sequences, the problem of model prediction error accumulation is alleviated. By ensuring the quality of the virtual sequence, the data utilization rate for model training can be improved, reducing the need for real samples, and thus improving the performance of the trained model.

[0010] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0011] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0012] Figure 1 A flowchart of a model training method for action sequence generation according to an embodiment of this application is shown; Figure 2 A schematic diagram of a model training method for action sequence generation according to an embodiment of this application is shown; Figure 3 A block diagram of a model training apparatus for action sequence generation according to an embodiment of this application is shown; Figure 4 A block diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0013] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0014] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0015] The following terms will be used in the following text: A graphical user interface (GUI) is a user interface design that allows users to interact with the system or software application through graphical elements such as windows, icons, menus, buttons, and text boxes. It allows users to interact with the system or software application through mouse clicks, keyboard input, touch operations, etc., without relying on complex command-line input.

[0016] In RPA (Robotic Process Automation) technology, an intelligent agent automatically completes work tasks by simulating the interaction between human users and software systems.

[0017] Interface elements are components in a graphical user interface used to enable interaction and display information. Through proper layout and design, they enable users to interact with software or devices intuitively.

[0018] The Vision-Language-Action (VLA) model is a multimodal artificial intelligence model that integrates visual perception, natural language understanding, and action execution.

[0019] Existing technologies for training action sequence generation models generally lack the construction of structural perturbations and bidirectional reasoning mechanisms for action sequences, resulting in insufficient generalization ability and robustness in complex scenarios. Current technologies typically train models directly using complete action sequences, making them prone to relying on surface-level patterns of fixed processes rather than learning the inherent logical connections between actions. For example, in automated testing scenarios, the model might only remember the order of "login → search → submit," but fail to understand that "login" is a necessary prerequisite for "submit." Because no action erasure mechanism is introduced, the model does not face the challenge of missing information, making it difficult to complete key actions through contextual reasoning when encountering missing actions or changes in their order (such as a user accidentally skipping a step) in real-world applications, easily leading to process interruptions or errors.

[0020] It should be noted that the application scenarios or examples provided in the embodiments of this application are for ease of understanding, and the embodiments of this application do not specifically limit the application of the technical solutions. In addition, the user information (including but not limited to user device information, user personal information, image information of the operation interface, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, user instruction data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with the relevant laws, regulations and standards of the relevant regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0021] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0022] Figure 1 A flowchart of a model training method for action sequence generation according to an embodiment of this application is shown, such as... Figure 1 As shown, the method may include steps S101, S102 and S103.

[0023] Step S101: Remove the target action from the initial action sequence to obtain the training sequence; Step S102: Based on the training sequence, use the target model to generate multiple action sequences with different execution orders; Step S103: According to the action differences of multiple action sequences, iteratively train the target model to obtain the trained target model.

[0024] For example, the aforementioned executing entity is an intelligent agent, and the aforementioned action sequence can be a programmed virtual action or a real action controlling a robot. In RPA scenarios, the intelligent agent acts as a virtual assistant for cross-system operations, understanding user needs through natural language processing and automatically operating multiple software interfaces to complete tasks. The aforementioned user instructions can be executed across systems. When a user instruction involves multiple systems, such as system A, email, and system B, the intelligent agent will generate an action sequence containing "login to system A → export data → send email → update status in system B". If the "send email" step is erased during training, the model needs to complete it based on the context (e.g., automatically triggering the email client after detecting the exported file). The intelligent agent relocates elements through visual analysis rather than relying on a fixed script. As a program execution method, it can also be applied in in-vehicle or smart home scenarios, where the intelligent agent acts as a multimodal interaction hub, integrating voice, gestures, and interface visual information.

[0025] In the Industrial Internet of Things (IIoT) or collaborative robots, intelligent agents act as a bridge between the physical and digital interfaces, coordinating virtual commands and physical actions. These physical actions can be monitored by sensors to track changes in the position of the robotic arm and determine the actual sequence of actions to be executed.

[0026] In some embodiments provided in this application, after removing the target action, the model needs to complete the action sequence through contextual reasoning. During training, key target actions are identified and removed from the initial action sequence. Target actions are considered to be those actions that have a significant impact on task completion and are among the identified node actions. By removing these target actions, the model learns how to infer the missing key actions from the context, thereby enhancing the model's contextual reasoning ability. Based on the training sequence obtained after removing the target actions, multiple action sequences with different execution orders are generated using the target model (such as the VLA model). For example, they can be generated through forward execution and backward backtracking, covering a wider state space, thereby improving the model's generalization ability and adaptability. The target model is iteratively trained according to the differences in actions of multiple action sequences, enabling the model to learn the implicit correlations between different execution orders, resulting in a well-trained target model that can effectively enhance the model's contextual reasoning ability and generalization ability.

[0027] According to some embodiments provided in this application, in order to identify which target nodes are worth removing before performing step S101, the method further includes the following specific steps: An action recognition model is used to identify node actions in the process actions included in the initial action sequence. The action recognition model is trained based on a reference action sequence marked with reference nodes. Node actions are actions that affect the completion state of reference instructions in multiple process actions. Reference instructions correspond to reference action sequences. Determine the target action based on the node actions.

[0028] In the embodiments provided in this application, the action recognition model is trained based on a reference action sequence labeled with reference nodes. The reference action sequence is an action sequence with key nodes labeled; key nodes can be considered actions that affect the completion state of a reference instruction. In the action sequence generation model, process actions and node actions are two different concepts. Process actions refer to all possible actions executed during the completion of a task, including key and non-key actions. Process actions are the specific operational steps in the task execution process, covering all actions from the start to the end of the task, regardless of whether these actions directly affect the final completion of the task. Node actions refer to key actions that significantly affect the task completion state during task execution. Node actions can be considered key steps for successful task completion, directly affecting the final result of the task. There can be one or more node actions, and the node actions corresponding to the initial action sequence for different operation types are different. The reference action sequence is a historical action sequence with labeled key nodes (i.e., reference nodes), and each reference node corresponds to an action that affects the completion state of a reference instruction. Through supervised learning training, the action recognition model takes all process actions included in the initial action sequence as input and outputs a certain number of node actions. The process of identifying node actions explicitly marks the key steps in the task, which helps in analyzing and understanding the model's decision-making logic.

[0029] For example, process actions include various types of operations, such as sliding the cursor, dragging the page, and scrolling the page. The aforementioned node actions can be actions marked with keywords including fields such as approval, payment, and submission, or actions that trigger cross-platform interactions.

[0030] According to some embodiments provided in this application, in step S101, when removing target actions from the initial action sequence, the number of target actions removed must match the inherent characteristics of the action sequence to avoid the model being unable to learn effectively due to excessive removal, or the model lacking reasoning challenge due to insufficient removal. This can be dynamically adjusted according to task characteristics and data distribution. For example, the removal ratio can be dynamically determined based on the total length of the initial action sequence; longer sequences allow for more removal, while shorter sequences retain more actions. Action sequences typically contain redundant or auxiliary steps, allowing for the removal of more target actions; shorter sequences, due to their compact structure, require the retention of more key actions to maintain task integrity. For example, if the initial sequence contains 20 actions (such as a complex approval process), a higher proportion (such as 40%-50%) of target actions can be removed, forcing the model to complete key steps through long-distance contextual reasoning. If the sequence contains only 5 actions (such as a simple login process), only 1-2 target actions are removed to avoid the model being unable to learn due to insufficient information.

[0031] The initial action sequence can be value-labeled, with each action representing its impact on the final instruction. Actions are ranked by value, prioritizing the removal of high-importance actions from the initial sequence as a way to strengthen high-value segments. Each action can be assigned a value score (1-5) using action recognition models or domain expert annotations; higher scores indicate greater impact on task completion. For example, in the "order takeout" node, "select payment method" and "confirm order" are high-value nodes, directly determining whether the order is generated; "swipe to view menu details" is a low-value node, affecting user experience but not the success of the task. The absence of high-value nodes significantly disrupts task integrity, forcing the model to supplement key steps through deep semantic understanding and contextual association. For example, if the high-value node "Confirm Order" is removed, the model needs to infer the missing confirmation action from the user's instruction "I want to place an order" and the unconfirmed order status displayed on the current page, rather than relying on fixed process memory. It can learn in advance how to backtrack from abnormal states and recover tasks, thus improving robustness.

[0032] Optionally, the two strategies described above can be combined. The first layer of filtering determines the total number of removable target actions based on sequence length and task complexity; for example, a maximum of 5 actions can be removed from long sequences, and a maximum of 2 from short sequences. The second layer of sorting sorts the nodes within the removable range from highest to lowest value, selecting a predetermined number of node actions from the total number of removable actions for removal. For example, for long sequences, 5, 4, 3, 2, and 1 nodes can be selectively removed. A combination of removing a certain number of high-value nodes and some low-value nodes can also be used, and this can be adjusted according to the specific application scenario.

[0033] Furthermore, model performance can be monitored using a validation set. If slow convergence or a decrease in prediction accuracy occurs, the removal ratio of high-value nodes can be automatically reduced to avoid excessive unsolvability of training data. For example, for a long sequence containing 15 process actions, which could be a business approval process, the maximum number of node actions that can be removed can be determined based on the proportion of each node action in the sequence length, such as a maximum of 5 node actions.

[0034] Node actions that can be removed are sorted by value, which is determined based on the action type and the task type. Node actions with a value greater than a predetermined threshold are selected to generate a first node set, and those with a value less than or equal to the predetermined threshold are selected to generate a second node set. A first number of node actions are selected from the first node set, and a second number from the second node set, to be removed as target nodes. If, during training, the model completes the node actions in the first node set and the accuracy of the completion result is lower than a predetermined accuracy threshold, it indicates that too many high-value action nodes have been removed, and the model's learning difficulty needs to be reduced. The first number of nodes selected from the first node set is reduced, and the second number selected from the second node set is increased. The target nodes are then reselected and removed. Training is then performed based on the reselected target nodes, gradually increasing the model's learning difficulty.

[0035] After sorting the five node actions by value, the top three high-value nodes, such as "salesperson submits for review", "supervisor's electronic signature", and "submit to system", and two low-value nodes, such as "slide page" and "zoom page", are selected as target nodes to be removed. If it is found that the accuracy of the model in completing high-value nodes is lower than the predetermined accuracy threshold of 50% during training, the user instruction of this type is adjusted to remove only two high-value nodes and three low-value nodes, gradually increasing the difficulty.

[0036] By reasonably controlling the number and priority of target actions removed, a balance is found between model learnability and reasoning challenge. Dynamically adjusting the number ensures that the training sequence has enough information for the model to learn, and can also stimulate reasoning ability through appropriate perturbation. Prioritizing the removal of high-value nodes focuses on improving the model's understanding of the core task logic. The combination of the two can significantly optimize the training efficiency and generalization ability of the action sequence generation model.

[0037] According to the embodiments provided in this application, the multiple action sequences include two sequences with opposite execution orders, namely, a forward action sequence and a reverse action sequence. In step S102: based on the training sequence, the target model is used to generate multiple action sequences with different execution orders, which may include the following steps: Based on the predicted direction from the preceding action to the following action in the training sequence, a forward action sequence is generated using the target model; the preceding action is the action that precedes the target action in the execution order, and the following action is the action that follows the target action in the execution order. Based on the predicted direction of the forward-driving action in the training sequence, the target model is used to generate the reverse action sequence.

[0038] In some embodiments provided in this application, the set of actions preceding the removed target action in the training sequence is considered as preceding actions. Action sequences are generated from preceding actions to subsequent actions, i.e., according to the natural execution order, predicting subsequent actions step by step from the current state. In the training sequence, the set of actions following the target action is considered as following actions, and action sequences are generated from following actions to preceding actions, i.e., in reverse order. Following actions are used as conditions input to the target model, which starts from the following state after the target action and infers the preceding actions in reverse. Forward action sequences are used to strengthen the model's understanding of sequential causal relationships, i.e., performing action A should lead to the execution of action B. Reverse action sequences are used to train the model to understand the implicit reverse relationship between actions, which can be understood as a necessary condition, i.e., to complete action B, action A must be executed first. This allows the model to verify the rationality of action logic from two directions, improving the accuracy of decision-making.

[0039] For example, in action sequence generation, in addition to the two basic directions of forward and backward, model prediction can also be performed through skip generation or branching directions. Skip generation can be understood as an interval prediction method, which skips several steps from the preceding action to directly predict subsequent actions further away, or predicts earlier preceding actions from the following action in reverse.

[0040] Branching directions generate multiple possible action sequence branches based on different conditions. These conditions can be configured according to system state or user preferences, and multiple branching directions can exist. Assuming the initial action sequence includes multiple branch sequences, when the target action is a branch point action, the model can generate multiple predicted sequences based on the preceding actions of the target action. Theoretically, the predicted sequences should correspond to the branch sequences of the corresponding branches. If the multiple predicted sequences generated by the model do not match the multiple branch sequences, a branch loss is generated for iterating the model.

[0041] Branch reinforcement learning models allow an agent to choose multiple actions in a given state, each action potentially triggering a new subsequent state. The decision history is no longer a linear sequence but a tree structure, capable of handling complex scenarios where multiple options coexist. When multiple predicted sequences lack complete action branches compared to multiple branch sequences, it is preferable to increase the weight of the penalty term. In action sequence generation for complex tasks involving multimodal and multi-path planning, the absence of complete action branches, i.e., the model has encountered an execution path ignoring problem, incurs a higher penalty than simple inaccurate action prediction. Assuming the target task has N branches and the model predicts M branches, if the number of predicted branches is less than the preset number (M < N), a branch penalty is triggered. The total penalty increases linearly or non-linearly as more branches are missing. Since not all branches are equally important, the penalty weight can be adjusted based on the semantic importance or probability of occurrence of the branches. If the frequency or probability of the missing action branch is greater than a predetermined threshold, the branch penalty is set to a predetermined weight. If the frequency or probability of the missing action branch is less than the predetermined threshold, the branch penalty is set to less than the predetermined weight, allowing the model to prune within a certain range to optimize computational efficiency.

[0042] Optionally, the above-mentioned branch point identification methods can be multiple. When an operation causes the system to enter different modes, the action node is identified as a branch point; when users with different permissions are presented with different operation options, the action node is identified as a branch node.

[0043] According to the embodiments provided in this application, iteratively training the target model based on the action differences of multiple action sequences to obtain a trained target model may include the following steps: Determine the first loss between the first action and the target action, and the second loss between the second action and the target action; the first action is the action predicted from the target action in the forward action sequence, and the second action is the action predicted from the target action in the reverse action sequence. Based on the first loss and the second loss, the joint loss is obtained; The target model is trained iteratively using joint loss to obtain a well-trained target model.

[0044] In the embodiments provided in this application, the maximum likelihood loss (Supervised Loss) method can be used for bidirectional loss calculation. The initial action sequence can be considered as having artificially taught data, where the erased target action is a real action with prior knowledge. If the model can know the erased target action during training, the deviation between the first action predicted by the model in the forward direction and the real target action should be as small as possible, that is, the predicted action distribution should be as close as possible to the real operation, thus obtaining the first loss representing the difference between the two. Similarly, the difference between the second action predicted by the model in the backward direction and the real target action is represented as the second loss. The first loss and the second loss can be combined according to weights to balance the importance of forward and backward inference, resulting in a joint loss.

[0045] By combining forward and backward prediction losses for iterative training, this embodiment enables the model to learn the dependencies between actions from a bidirectional perspective, adapting to different execution orders or abnormal interruptions.

[0046] For example, the weights can be dynamically adjusted according to the training phase. For instance, in the early stages of training, the weight of the first loss can be increased while the weight of the second loss can be decreased, allowing the model to prioritize learning positive causal relationships. In the later stages of training, the weight of the first loss can be decreased while the weight of the second loss can be increased, strengthening the learning of inverse necessary conditions.

[0047] According to the embodiments provided in this application, iteratively training the target model based on the action differences of multiple action sequences to obtain a trained target model may include the following steps: The cross-validation loss is determined based on the difference between the first action and the second action; the first action is the action predicted from the target action in the forward action sequence, and the second action is the action predicted from the target action in the reverse action sequence. Cross-validation loss is used to iteratively train the target model to obtain a well-trained target model.

[0048] In the embodiments provided in this application, a self-supervised learning mechanism is formed by cross-validation loss, which utilizes the prediction results of forward and backward action sequences to mutually verify each other. The consistency of the model's predictions is measured by the difference between the target action predicted by the forward action sequence (first action) and the target action predicted by the backward action sequence (second action). If the model's forward and backward predictions for the same target action are consistent, it indicates that the model's understanding of the dependencies between actions is more reliable; otherwise, there is a logical contradiction that needs optimization. The cross-validation loss is backpropagated to the model to adjust the model parameters to reduce the difference between the forward and backward prediction results, enabling the model to learn consistent action dependency logic.

[0049] For example, there are several ways to calculate the difference between the first and second actions. This can be achieved through classification or regression tasks. When actions are represented as discrete categories (such as "clicking a button" or "submitting a form"), a probability distribution difference metric can be used. This can be achieved by calculating the KL (Kullback-Leibler Divergence) divergence or the JS (Jensen-Shannon Divergence) divergence between the first and second actions. For instance, the KL divergence measures the difference between two probability distributions, reflecting the information loss from one distribution to another. For example, if the probability of a positive prediction of "payment" is 0.8, and the probability of a negative prediction is 0.6, then the KL divergence quantifies this probability difference, guiding the model to unify the prediction results.

[0050] When actions are represented as continuous values ​​(such as coordinates or time intervals), numerical difference measures can be used to calculate the absolute error or Euclidean distance between the first and second actions. The absolute error is calculated as the average of the absolute values ​​of the differences between the predicted and actual values, making it more robust to identifying action differences. Since the predicted first and second actions are derived from the target action, the predicted and actual actions have the same feature dimensions. The Euclidean distance directly reflects the straight-line distance between two actions in high-dimensional space.

[0051] According to some embodiments provided in this application, generating multiple action sequences with different execution orders based on training sequences using a target model may include the following specific steps: Based on the interface screenshots corresponding to the process actions included in the training sequence, the historical semantic features corresponding to the historical instructions, and the training sequence, multiple action sequences are generated using the target model; the historical semantic features correspond to the initial action sequence.

[0052] In the embodiments provided in this application, multimodal information and action sequences with execution order are fused for correspondence. Each process action in the action sequence has corresponding visual features and semantic features. The visual features are the interface screenshots in the execution state, and the semantic features are extracted based on user commands. Through the above modal correspondence processing, the corresponding interface screenshots and historical commands are labeled for each action sequence. By fusing interface screenshots, historical semantic features and action sequences, the model can learn more comprehensive information.

[0053] The target model obtained through any of the above embodiments can achieve good action backtracking capabilities, adjusting the execution path in real time through multimodal information to ensure the completion of user instructions. The method further includes the following steps: Get the first real-time screenshot of the operation interface, as well as the current semantic features corresponding to the user's command; Based on the first real-time screenshot and the current semantic features, the trained target model is used for processing to determine the first action sequence; In response to the action backtracking of any third action in the first action sequence, the trained target model is used to process the action backtracking from the third action to the fourth action based on the second real-time screenshot and the current semantic features; the second real-time screenshot is a screenshot of the operation interface when the third action has been executed. Based on the third real-time screenshot and the current semantic features, the trained target model is used for processing to generate a second action sequence starting from the fourth action; the third real-time screenshot is a screenshot of the operation interface when the fourth action is executed, and the second action sequence is used to complete the user's instructions.

[0054] In the embodiments provided in this application, a real-time screenshot of the current operation interface is obtained as visual input, and semantic features corresponding to the user's command are extracted as semantic input. The first real-time screenshot and the current semantic features are input into a trained target model. The target model generates a first action sequence based on the input visual and semantic information to execute the user's command. In the first action sequence, assuming that any third action is executed, the action is executed and backtracking is performed. After executing the third action, a real-time screenshot of the operation interface is obtained, i.e., a second real-time screenshot. Based on the second real-time screenshot and the current semantic features, the trained target model is used for processing to execute the operation of backtracking from the third action to the fourth action. The fourth action is the operation action before the third action, which may be adjacent to or not adjacent to the third action. This means that the target model needs to reverse-generate the fourth action, which should be executed, based on the current state and the user command. The system backtracks from the current state (after the third action) to the fourth action and updates the interface state. After executing the fourth action, a third real-time screenshot of the operation interface is obtained. Based on the third real-time screenshot and the current semantic features, the trained target model generates a new action sequence as the second action sequence. Starting with the fourth action, the sequence of actions continues according to the second action until the user instruction is completed. Through this process, the target model achieves good action backtracking capability, ensuring that it can automatically adjust and correct for problems encountered during execution, thereby improving the success rate and efficiency of task execution.

[0055] For example, the triggering conditions for the above-mentioned action backtracking can be various, including user-initiated backtracking or automatic system detection. A user may be dissatisfied with a certain operation (such as the third action) in the current action sequence and trigger a backtracking request. The intelligent agent provides a human intervention interface, allowing users to cancel and backtrack specified actions through point-and-click commands or voice commands.

[0056] For the automatic detection method, the model detects an abnormal interface state after executing the third action (such as an error message) and proactively triggers backtracking. Interface state abnormalities can take various forms, such as error pop-ups, element failures, and system performance exceeding limits. OCR (Optical Character Recognition) can be used to identify keywords like "error" and "failure," and if the confidence level is greater than a predetermined threshold, backtracking is triggered. For element failures, the operability of controls or interface elements can be checked; if there is no response after more than a predetermined number of consecutive operations (e.g., 3 times), proactive backtracking is triggered. For performance exceeding limits, a certain percentile value can be used to determine if the single-step operation time exceeds the historical required time. For example, exceeding the historical 95th percentile value. This is considered as sorting the time of the past 100 similar operations, with the 95th largest value being the 95th percentile value, meaning that 95% of historical operations were faster than the current single-step operation.

[0057] Based on the above embodiments and optional embodiments, this application also provides an optional implementation method: a visual language action (VLA) model training method based on bidirectional reinforcement learning. This method introduces forward and backward models and policies (i.e., a bidirectional trajectory generation mechanism) into the traditional large VLA model framework. Through continuous iterative iteration across three stages—data acquisition, model training, and policy optimization—efficient and robust multimodal end-to-end learning is achieved. The trained target model is used to help users quickly find the products they need on an e-commerce platform for car wash products. Users can input their needs via voice or text, such as "finding a car windshield washer fluid suitable for winter use." The agent needs to understand the user's needs and navigate to the correct category page on the platform through a series of actions, ultimately displaying products that meet the criteria.

[0058] During the target model training phase, an action recognition model is first used to identify key node actions in the initial action sequence. This model is trained based on a reference action sequence labeled with reference nodes. For example, in a car wash product category navigation task, the reference action sequence includes steps such as "opening the homepage → searching for products → filtering categories → selecting products," where "filtering categories" is a key node action and serves as the target action.

[0059] Based on the output of the action recognition model, the target actions to be removed are determined. Removing the "filtering and classification" target action requires the model to complete this missing step through contextual reasoning during training. On one hand, a normal action sequence is generated, starting from the initial action of the training sequence and generating the action sequence in the natural execution order. For example, starting with "opening the homepage," subsequent actions are predicted step by step until the task is completed. On the other hand, a reverse action sequence is generated, starting from the final action of the training sequence and generating the action sequence in reverse order. For example, starting with "selecting a product," the preceding actions are reasoned backward until the starting state is returned.

[0060] The model policy parameters are updated end-to-end using a combination of self-supervised learning and maximum likelihood criterion, combining positive and negative samples. For the maximum likelihood criterion approach, the differences between the predicted and target actions in the forward and reverse action sequences are determined, and a first loss and a second loss are calculated. For example, the difference between the forward predicted "screening and classification" action and the actual action is represented by the first loss, and the difference in the reverse prediction is represented by the second loss. The first and second losses are combined by weights to obtain the joint loss. The target model is iteratively trained using the joint loss, enabling the model to learn the dependencies between actions from a bidirectional perspective. For the self-supervised learning approach, the prediction results of the forward and reverse action sequences are cross-validated, and the model parameters are adjusted using cross-validation loss, forcing the model to learn consistent action dependency logic.

[0061] After training the target model in the above manner, in practical applications, the system obtains the first real-time screenshot of the current operation interface, as well as the current semantic features corresponding to the user's command. Figure 2 A schematic diagram of a model training method for action sequence generation according to an embodiment of this application is shown, such as... Figure 2 As shown, the task is to "find a car windshield washer fluid suitable for winter use." The agent extracts the semantic features "winter" and "car windshield washer fluid." The first real-time screenshot and the current semantic features are input into the trained target model to generate the first action sequence. For example, "Open homepage → Search for products → Filter categories → Select product." In the historical action sequences used for training the target model, each node action is labeled with its action type, the coordinates of the executed operation, and the reasoning and thought process of the target model for that step.

[0062] Suppose an issue occurs during the "filtering and categorizing" (third action) step, triggering a backtracking mechanism. A second real-time screenshot is captured after the third action. Based on this screenshot and the current semantic features, the target model is used to perform the backtracking operation from the third action to the fourth action. For example, the model might determine that a new "product search" is needed first.

[0063] After executing the fourth action, a third real-time screenshot is acquired. Based on the third real-time screenshot and the current semantic features, a new action sequence (the second action sequence) is generated. This new action sequence continues to complete the user's instructions, starting from the fourth action. For example, "Re-search for products → Re-filter categories → Select products".

[0064] Through the above implementation methods, the target model achieves excellent action backtracking capabilities, ensuring automatic adjustment and correction when problems are encountered during execution. The model is trained by fusing multimodal information (visual and semantic features) and action sequences with different execution orders, enhancing its contextual reasoning and generalization abilities. In practical applications, the system can dynamically adjust the execution path based on real-time screenshots and user commands, improving the success rate and efficiency of task execution.

[0065] The optional implementation methods provided in this application significantly improve the data utilization rate of VLA model training, reduce the need for actual samples, and are suitable for scenarios with scarce training resources. By collaborating with forward and backward models, the accumulation of model errors is greatly suppressed, resulting in a significant improvement in the quality of virtual simulation samples and more stable and reliable training. The entire multimodal perception, reasoning, and decision-making chain is optimized, accelerating the generalization and transfer capabilities of large VLA models in complex scenarios. These processing methods can be extended to various practical application scenarios such as complex robot automation, recommendation systems, autonomous driving, and intelligent manufacturing, providing an efficient theoretical and engineering foundation for multimodal AI closed-loop decision-making.

[0066] Figure 3 A block diagram of a model training apparatus for action sequence generation according to an embodiment of this application is shown, such as... Figure 3 As shown, corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide a model training device for action sequence generation, including: The action removal module 301 is used to remove the target action included in the initial action sequence to obtain the training sequence; Action sequence generation module 302 is used to generate multiple action sequences with different execution orders based on training sequences and using a target model; The model training module 303 is used to iteratively train the target model according to the action differences of multiple action sequences to obtain a trained target model.

[0067] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0068] Figure 4 This is a block diagram of an electronic device used to implement embodiments of this application. For example... Figure 4As shown, the electronic device includes a memory 401 and a processor 402. The memory 401 stores a computer program that can run on the processor 402. When the processor 402 executes the computer program, it implements the method described in the above embodiments. The number of memories 401 and processors 402 can be one or more. In a specific implementation, the electronic device may also include a communication interface 403 for communicating with external devices and performing data exchange and transmission.

[0069] In practical implementation, if the memory 401, processor 402, and communication interface 403 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0070] Optionally, in a specific implementation, if the memory 401, processor 402 and communication interface 403 are integrated on a single chip, the memory 401, processor 402 and communication interface 403 can communicate with each other through an internal interface.

[0071] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0072] This application provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in this application.

[0073] This application also provides a chip, which includes a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0074] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0075] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0076] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Sync Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0077] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0078] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0079] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0080] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0081] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0082] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0083] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0084] The above are merely exemplary embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A model training method for action sequence generation, characterized in that, include: Remove the target action from the initial action sequence to obtain the training sequence; Based on the training sequence, the target model is used to generate multiple action sequences with different execution orders; Based on the differences in the actions of the multiple action sequences, the target model is iteratively trained to obtain the trained target model.

2. The method according to claim 1, characterized in that, The multiple action sequences include forward action sequences and reverse action sequences. The generation of multiple action sequences with different execution orders based on the training sequences using the target model includes: Based on the predicted direction from the preceding driving action to the following driving action in the training sequence, a forward action sequence is generated using the target model; the preceding driving action is the action before the target action in the execution order, and the following driving action is the action after the target action in the execution order. Based on the predicted direction from the subsequent driving action to the preceding driving action in the training sequence, the target model is used to generate a reverse action sequence.

3. The method according to claim 2, characterized in that, The step of iteratively training the target model according to the action differences of the multiple action sequences to obtain the trained target model includes: Determine a first loss between a first action and the target action, and a second loss between a second action and the target action; the first action is the action predicted from the target action in the forward action sequence, and the second action is the action predicted from the target action in the reverse action sequence. Based on the first loss and the second loss, the joint loss is obtained; The target model is iteratively trained using the joint loss to obtain the trained target model.

4. The method according to claim 2, characterized in that, The step of iteratively training the target model according to the action differences of the multiple action sequences to obtain the trained target model includes: Based on the difference between the first action and the second action, the cross-validation loss is determined; the first action is the action predicted for the target action in the forward action sequence, and the second action is the action predicted for the target action in the reverse action sequence. The target model is iteratively trained using the cross-validation loss to obtain the trained target model.

5. The method according to claim 1, characterized in that, The method includes: An action recognition model is used to identify node actions in the process actions included in the initial action sequence. The action recognition model is trained based on a reference action sequence marked with reference nodes. The node action is an action that affects the completion state of a reference instruction among multiple process actions. The reference instruction corresponds to the reference action sequence. The target action is determined based on the node action.

6. The method according to claim 1, characterized in that, The step of generating multiple action sequences with different execution orders using the target model based on the training sequence includes: Based on the interface screenshots corresponding to the process actions included in the training sequence, the historical semantic features corresponding to the historical instructions, and the training sequence, the target model is used to generate the multiple action sequences; the historical semantic features correspond to the initial action sequence.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: Get the first real-time screenshot of the operation interface, as well as the current semantic features corresponding to the user's command; Based on the first real-time screenshot and the current semantic features, the trained target model is used for processing to determine the first action sequence; In response to performing action backtracking on any third action in the first action sequence, the trained target model is used for processing based on the second real-time screenshot and the current semantic features to execute the backtracking from the third action to the fourth action; the second real-time screenshot is a screenshot of the operation interface when the third action has been executed; Based on the third real-time screenshot and the current semantic features, the trained target model is used for processing to generate a second action sequence starting from the fourth action; the third real-time screenshot is a screenshot of the operation interface when the fourth action is executed, and the second action sequence is used to complete the user command.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

10. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images