Action sequence execution method, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention provides an action sequence execution method, electronic equipment, a storage medium and a program product. The method comprises the steps that a global action sequence is generated based on semantic features representing user instructions, and the global action sequence comprises a plurality of node actions with an execution sequence; after executing each node action, performing anomaly identification on each node action to obtain an identification result corresponding to each node action; generating a target action sequence taking the target node action as a starting point under the condition that the identification result indicates that the target node action is abnormal; the target action sequence is used for updating the global action sequence; according to the updated global action sequence, interface elements in the operation interface are processed until the execution target of the user instruction is completed. According to the technical scheme of the embodiment of the invention, the technical problem that the success rate of the automatic execution process is not ideal in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of human-computer interaction technology, and in particular to an action sequence execution method, electronic device, storage medium, and program product. Background Art

[0002] With the development of intelligent automation and human-computer interaction technologies, more and more intelligent agents are required to complete complex tasks through graphical user interfaces (Web GUIs), such as automated testing, automated form filling, and automated web page operation processes. Existing vision- and language-based intelligent agents still face significant bottlenecks in multi-step operations, multiple action sequences, and complex process control.

[0003] Related technologies use single-step operations with either coarse or fine granularity, resulting in low efficiency in complex tasks and difficulty forming high-level abstract actions. Existing solutions also lack effective modeling and reasoning for long action sequences or multi-step combinations, resulting in insufficient reliability and flexibility in task execution. They struggle to cope with large-scale, dynamically changing interfaces and suffer from limitations in the continuous coordination of multiple actions, limiting the generalization and practical value of automated agents. Summary of the Invention

[0004] The embodiments of the present application provide an action sequence execution method, electronic device, storage medium, and program product to alleviate or solve the technical problem of unsatisfactory success rate of automated execution process in related technologies.

[0005] In a first aspect, an embodiment of the present application provides an action sequence execution method, comprising: Based on the semantic features representing the user instructions, a global action sequence is generated, where the global action sequence includes multiple node actions with an execution order. After executing each node action, perform abnormal identification on each node action to obtain the identification result corresponding to each node action; When the recognition result indicates that the target node action is abnormal, a target action sequence is generated starting from the target node action; the target action sequence is used to update the global action sequence; According to the updated global action sequence, the interface elements in the operation interface are processed until the execution target of the user instruction is completed.

[0006] In a second aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any method of the embodiment of the present application when executing the computer program.

[0007] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the method of any one of the embodiments of the present application is implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements any method of the embodiments of the present application when executed by a processor.

[0009] Based on the action sequence execution method of the first aspect above, the present application has at least the following beneficial effects or advantages: Through a semantically driven intelligent planning and dynamic adjustment mechanism, a closed-loop system with self-correction capabilities is constructed. After each operation, the system monitors the execution status in real time through multimodal perception (visual recognition, interface structure analysis, and semantic understanding). It can accurately identify various abnormal situations such as element occlusion and state timeouts. When an abnormal situation is detected, the system does not simply terminate the process, but intelligently generates targeted correction solutions. For example, when it finds that an interface element is blocked, it automatically adjusts and increases the operation action to eliminate the occlusion effect. This dynamic adjustment capability enables the system to flexibly respond to various interface changes and unexpected situations, significantly improving the smooth completion rate of the automated process. Compared with traditional automation technology, the embodiment of the present application greatly reduces process interruptions caused by unexpected situations. At the same time, through intelligent local optimization strategies, exception handling is made more efficient and smooth. This dynamic adjustment capability enables the system to adapt to various complex and dynamic environments and improve the robustness of task execution.

[0010] The above description is only an overview of the technical solution of this application. In order to more clearly understand the technical means of this application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of this application more obvious and easy to understand, the specific implementation methods of this application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar components or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments according to the present application and should not be regarded as limiting the scope of the present application.

[0012] Figure 1 A flowchart showing a method for executing an action sequence according to an embodiment of the present application is shown; Figure 2 A schematic diagram showing an action sequence execution method according to an embodiment of the present application is shown; Figure 3 A block diagram of an action sequence execution device according to an embodiment of the present application is shown; Figure 4 A block diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0013] Hereinafter, only certain exemplary embodiments are briefly described. As will be appreciated by those skilled in the art, the described embodiments may be modified in various ways without departing from the spirit or scope of the present application. Therefore, the drawings and description are to be regarded as illustrative in nature and not restrictive.

[0014] To facilitate understanding of the technical solutions of the embodiments of the present application, the following describes the related technologies of the embodiments of the present application. The following related technologies can be combined with the technical solutions of the embodiments of the present application as optional solutions, and all of them fall within the scope of protection of the embodiments of the present application.

[0015] The following terms will be used in the following text: A graphical user interface (GUI) is an interface design that uses graphical elements (such as windows, icons, menus, buttons, and text boxes) to interact with the user. It allows users to interact with computer systems or software applications through mouse clicks, keyboard input, and touch operations, without relying on complex command line input.

[0016] Interface elements are the components of a graphical user interface used to achieve interaction and display information. Through reasonable layout and design, users can interact with software or devices intuitively.

[0017] In RPA (Robotic Process Automation) technology, agents automatically complete work tasks by simulating the interaction between human users and software systems.

[0018] The Vision-Language-Action (VLA) model is a multimodal artificial intelligence model that integrates visual perception, natural language understanding, and action execution.

[0019] It should be noted that the above-mentioned application scenarios or application examples provided in the embodiments of this application are for ease of understanding, and the embodiments of this application do not specifically limit the application of the technical solution. In addition, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation portals for users to choose to authorize or refuse.

[0020] Existing technologies lack effective modeling and reasoning for long action sequences or multi-step combinations, are easily affected by factors such as different interface layouts, styles, and dynamic changes, have weak generalization capabilities, and have limited adaptability.

[0021] The following describes in detail the technical solution of this application and how it solves the aforementioned technical problems using specific embodiments. The several specific embodiments listed can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. The following describes the embodiments of this application in detail with reference to the accompanying drawings.

[0022] Figure 1 A flow chart of the method for executing an action sequence according to an embodiment of the present application is shown. Figure 1 As shown, the method may include steps S101 to S104.

[0023] Step S101: generating a global action sequence based on semantic features representing user instructions, wherein the global action sequence includes multiple node actions having an execution order; Step S102: After executing each node action, perform abnormality identification on each node action to obtain an identification result corresponding to each node action; Step S103: if the recognition result indicates that the target node action is abnormal, generating a target action sequence starting from the target node action; the target action sequence is used to update the global action sequence; Step S104: according to the updated global action sequence, the interface elements in the operation interface are processed until the execution target of the user instruction is completed.

[0024] For example, the execution entity is an agent, which can be a multi-action combination VLA (Vision-Language-Action) model for Web GUIs or a multimodal Transformer model. This agent, designed for scenarios such as page operation automation and GUI agents, uses an end-to-end vision, language, and action joint modeling approach to achieve multi-action sequenced control of complex web pages. Possible application scenarios include software robots (RPA robots), browser plug-ins, application extensions, intelligent interactive systems, Internet of Things (IoT) device controllers, and collaborative robots.

[0025] In the case of software robots (RPA robots), browser plug-ins, and application extensions, interactive intelligent agents facing end users understand user needs through natural language processing and operate the interface to complete services. The anomalies of the above node actions can include various types, such as: sudden pop-up windows in the operation interface (such as system upgrade notifications), verification code recognition errors, account lock prompts, the operating system or interactive interface elements are in an unresponsive state, dynamically loaded content does not appear on time, session timeout resulting in loss of operation permissions, operation frequency limit is triggered, insufficient memory causes application crashes, network delays cause response timeouts, etc.

[0026] The above-mentioned intelligent interactive system can be a smart cockpit, smart home central control, or an interactive hub integrated into a hardware device (such as an in-vehicle central control screen or a smart home panel). When used as an intelligent interactive system, an Internet of Things (IoT) device controller, or a collaborative robot, the above-mentioned action node anomalies may include various types, such as: sudden device offline prompts, communication interruption alarms, device function abnormality prompts, clock asynchrony between devices, manual interruption, etc.

[0027] In the embodiments provided herein, the semantic features of user instructions are parsed, and based on the semantic understanding of the instruction intent, a global execution sequence consisting of multiple ordered node actions is automatically generated. These node actions form a complete operation chain according to business logic. After completing each node action, the execution results are monitored in real time and anomalies are identified, including but not limited to interface element status verification, operation feedback analysis, and expected effect comparison. Identifying anomalies after executing a node action can promptly identify problems that may arise during execution, enhancing the stability and reliability of the system and preventing the failure of the entire operation or incorrect results due to abnormal conditions. When an anomaly is detected in a target node action, the process is not simply terminated. Instead, the cause of the anomaly is intelligently analyzed and a new action sequence, namely the target action sequence, is generated based on the current context, starting from the abnormal node. This target action sequence is used as a local correction solution to dynamically update the original global execution plan. The introduction of a dynamic monitoring and replanning mechanism enables the system to have autonomous adaptability, enabling it to flexibly respond to various unexpected abnormal situations, significantly improving the robustness of the process. Subsequent operations are continued according to the updated action sequence until the target task set by the user instruction is fully achieved.

[0028] According to the embodiment provided by the present application, in step S101: generating a global action sequence based on the semantic features representing the user instruction may include the following specific steps: Get the initial screenshot of the operation interface; Determine the action execution strategy based on the initial screenshot, semantic features, and historical action sequences. The historical action sequences are obtained based on the historical node actions that completed historical instructions. According to the action execution strategy, reference actions are selected in the predetermined action space to generate a global action sequence.

[0029] In the embodiments provided in this application, an initial screenshot of the operating interface is obtained as a visual reference. This is combined with intent recognition and parameter extraction of the semantic features of user commands, and with reference to prior knowledge of historical action sequences. The historical action sequences provide successful action patterns from historical execution records, and the optimal action execution strategy is determined through triple information fusion. The accumulation of historical action sequences forms a reusable knowledge base, facilitating the agent's continuous learning and optimization of action strategies. As the number of uses increases, the execution success rate and efficiency will continue to improve. By analyzing the distribution of interface elements in the initial screenshot, understanding the core demands of the semantic features, and drawing on effective experience from historical action sequences, the agent is provided with multi-dimensional information as a basis for decision-making, thereby obtaining an execution strategy. Based on the execution strategy, the reference action that best suits the current context is intelligently selected from a predefined action space (such as basic operations such as clicking, inputting, and scrolling), ultimately generating a structured global action sequence. The global action sequence not only contains the necessary operation nodes, but also clarifies the temporal execution order between each node.

[0030] According to the embodiment provided by this application, the method further includes: For predetermined type operations, extract multiple reference actions; An action space is generated according to the action types corresponding to the multiple reference actions, the positioning of each interface element, and / or the reference filled content.

[0031] In the embodiments provided in the present application, for specific types of operations (such as form filling, data query, navigation switching, etc.), the intelligent agent pre-collects and organizes a series of representative reference actions. The above-mentioned reference actions can come from historical execution records, manually annotated cases, or the experience of domain experts. Based on the extracted reference actions, the intelligent agent performs structured organization. Each reference action includes information in multiple dimensions, including the basic type of operation (such as click, input, selection, scrolling, etc.), the position information of the interface element that records the action in the operation interface, and for actions that require data input (such as form filling), typical input examples or templates are provided. By combining the information of the above dimensions, an action space containing possible actions and their parameters is generated, providing a candidate set for the subsequent generation of a global action sequence. The structured action space can integrate the experience and knowledge of domain experts, provide a clear set of action candidates, reduce the exploration space, and accelerate the model convergence speed and optimization efficiency.

[0032] For example, the input to the agent model includes a GUI screenshot of the current page (i.e., the user interface) as visual feature input, and a historical action sequence and a textual description of the current task as language feature input. The agent model can be a multimodal Transformer or a large-scale vision-language-action model, capable of cross-modal feature alignment and action reasoning, and can understand the visual features of interface elements, textual semantics, and action intentions.

[0033] The agent outputs the selected reference actions in the action space, resulting in a standardized sequence of combined actions, which then generates a global action sequence. For example, click, double-click, drag, input, scroll, and shortcut key operations are all expressed as "action type + coordinates / content" and have a temporal structure.

[0034] For example, there may be multiple common actions based on web page operations. The reference actions abstracted from the above action space may include: click (start_box='(x1,y1)'), which means to perform a left mouse click at the specified coordinates (x1,y1). start_box represents the coordinates of the click position in the format of the string '(x1,y1)', which is used to trigger interactive elements such as buttons, links, and drop-down menus.

[0035] doubleclick (start_box='(x1,y1)') means to double-click the left mouse button at the specified coordinates (x1,y1). The start_box represents the coordinates of the double-click position in the string '(x1,y1)'. It is used to open files, edit text, or trigger elements that require double-clicking to activate.

[0036] right_single (start_box='(x1,y1)') means to perform a right-click at the specified coordinates (x1,y1). start_box is used for the coordinates of the right-click position. The format is the string '(x1,y1)', which is used to open the context menu or trigger the right-click menu options.

[0037] drag (start_box='(x1,y1)', end_box='(x2,y2)') means dragging from the starting coordinate (x1,y1) to the target coordinate (x2,y2). start_box represents the coordinates of the starting point of the drag, and the format is the string '(x1,y1)'. end_box represents the coordinates of the end point of the drag, and the format is the string '(x2,y2)'. It is used for drag sorting, adjusting the size of elements, or performing operations that require dragging (such as slider control).

[0038] hotkey (key='') can execute keyboard shortcuts. key represents a hotkey in the form of a string. It quickly executes system or application functions and replaces mouse operations. For example, the shortcut key for refreshing the page is F5, and the shortcut key for copying and pasting is Ctrl + C / V.

[0039] type (content='', start_box='(x1,y1)') means entering text content at the specified coordinates (x1,y1). content represents the text content to be entered, in the format of a string. start_box represents the coordinates of the input location, in the format of the string '(x1,y1)'. It is used to fill in information in input boxes, text areas, and other locations.

[0040] scrolldown() means scrolling the page down, scrollup() means scrolling the page up, scrollleft() means scrolling the page to the left, and scrollright() means scrolling the page to the right. It is used to browse long pages.

[0041] scrollmenu (start_box='(x1,y1,x2,y2)') indicates scrolling within the specified area (x1,y1,x2,y2) (such as a drop-down menu or list). start_box indicates the coordinates of the upper left and lower right corners of the scrolling area. The format is the string '(x1,y1,x2,y2)'. It is used to browse content within a limited area (such as a selector or scrolling list).

[0042] finish() is used to mark the task completed and terminate the current operation sequence.

[0043] wait() is used to pause execution and wait for specific conditions, such as interface element loading or system response waiting time.

[0044] call_user() is used to pause and request user intervention, such as entering a verification code, allowing the user to select preferred content, or performing facial recognition to obtain operation permissions.

[0045] stop (reason='') terminates the task for the specified reason. If an error occurs, you can log the operation by recording the text in the reason parameter and attaching the current execution status (such as executed actions, current interface screenshots, and variable values) to facilitate subsequent troubleshooting.

[0046] The above reference actions can be combined into an ordered sequence to implement complex high-level task flows. To facilitate understanding, the task of searching for an item is used as an example: click('(100,200)') / / Click the search box; type('Something', '(100,200)') / / Enter the search content; click('(300,200)') / / Click the search button; scrolldown() / / Scroll down the page; wait() / / Wait for search results to load; click('(200,400)') / / Click on the first search result; finish() / / Complete the task.

[0047] In the above embodiment, the positioning of the interface elements is expressed in the form of coordinates. In order to improve the recognition capability of the interface elements, the positioning of the interface elements can also be in the form of an area frame.

[0048] According to the embodiment provided in this application, the method may further include the specific steps of: Based on the visual features of the operation interface and the semantic features of the user instructions, the fusion features are obtained; Based on the fusion features, a region box corresponding to each interface element is generated as a positioning, and the region box is used to represent the interactive area of ​​the corresponding interface element.

[0049] In the embodiment provided in this application, the visual features of the operating interface are extracted by computer vision technology, including but not limited to underlying features such as color distribution, texture pattern, and shape contour. At the same time, the two types of heterogeneous features are fused together to obtain fused features in combination with the natural language processing results of the semantic feature analysis of the user instructions. Based on this fused feature, the intelligent agent intelligently identifies the interactive elements in the interface and generates interactive boundary annotations for each interface element, that is, obtains the area box of each interface element. The above area box not only marks the physical position of the interface element in the screen coordinate system, but also selects the interactive range. Clicking on any one of the coordinates can successfully realize the interactive processing of the interface element. It has good adaptability to the identification requirements of interface elements with smaller icons in the operating interface, and improves the robustness of the model.

[0050] By analyzing the visual features of the user interface (such as element shape, color, texture, and layout) and the semantic features of user commands (such as keywords and intent categories), the system fuses these two elements to generate a more comprehensive fusion feature. This system can associate semantic intent with visual elements in the interface, for example, linking the semantics of "clicking the search button" with the visual area of ​​the search icon or "Search" text label in the interface. This allows it to maintain stable recognition accuracy even when the interface style or elements change suddenly.

[0051] For example, visual features can be used to extract high-dimensional vectors through convolutional neural networks, semantic features can be encoded into semantic space representations through language models, and then cross-modal alignment can be achieved through the feature cross-attention mechanism to finally generate a fused feature representation.

[0052] According to some embodiments provided by this application, generating a region frame corresponding to each interface element as a positioning based on the fusion features may include the following steps: Based on the fusion features, predict the initial corner coordinates of each interface element; Normalize the initial corner coordinates of each interface element to obtain the target corner coordinates of each interface element. There are at least two target corner coordinates, which are used to anchor the range of the area frame.

[0053] In the embodiment provided in the present application, the output layer of the intelligent model includes multiple regression heads, and every two regression heads are specifically responsible for predicting the initial coordinates of a type of corner point. Since one coordinate includes the horizontal coordinate x and the vertical coordinate y, for a rectangular area box, there are two types of corner points (upper left and lower right), so there will be two regression heads for predicting the horizontal and vertical coordinates of the upper left corner point. Similarly, there will be two regression heads for predicting the lower right corner point, allowing the model to learn the features of each corner point separately, thereby improving the accuracy of the prediction. Each regression head predicts the initial coordinates of the corresponding corner point based on the fusion features of the input. The above-mentioned initial corner point coordinates can be regarded as the original prediction value without normalization. The normalization process makes the output of the model independent of the specific size of the input image, thereby improving the generalization ability of the model. According to the normalized multiple target corner point coordinates, the area box of each interface element is constructed. The target corner point coordinates define the shape and position of the area box, thereby selecting the interactive area of ​​the current interface element.

[0054] For example, a fully connected layer is provided in the agent model to further map the initial corner point coordinates of each corner point to the final target corner point coordinates. During the mapping process, the fully connected layer normalizes the initial corner point coordinates to a predetermined size range.

[0055] According to some embodiments provided herein, a global action sequence is composed of process actions, and multiple node actions belong to the process actions included in the global action sequence. Among the actions in the entire process, there are some actions that play a key role in implementing user instructions, namely node actions. For the multiple node actions included in the global action sequence in step S101, their selection method can be implemented by the following steps, and the method also includes: A node recognition model is used to determine the process actions used to change the execution state among multiple process actions as multiple node actions. The node recognition model is trained based on multiple historical action sequences with labeled historical nodes.

[0056] In the embodiment provided in the present application, a node recognition model is obtained by pre-training multiple historical action sequences based on historical nodes, and multiple process actions contained in the global action sequence are analyzed to identify process actions that can change the execution state and determine them as multiple node actions. These node actions are the core steps for implementing user instructions in the global action sequence. Key nodes are screened from process actions through model training to ensure the effectiveness and pertinence of subsequent action anomaly identification. By eliminating non-critical process actions and streamlining the global action sequence, the number of actions that need to be monitored and analyzed is reduced, the complexity of subsequent anomaly identification is reduced, and the abnormal calculation overhead during the action process is reduced.

[0057] Exemplarily, the above-mentioned node recognition model can be continuously iterated and updated based on historical data, and the recognition rules of node actions can be continuously optimized according to a predetermined period, so that the global action sequence can be dynamically adjusted as the business scenario changes. The above-mentioned business scenario changes can be expressed as changes in business rules, including interface revisions and process adjustments, to maintain the ability to efficiently execute user instructions.

[0058] According to some embodiments provided herein, based on the heterogeneity of node actions, different node actions can be classified. Multiple node actions correspond to different abnormal patterns, and subsequent abnormality identification processing can be performed on the node actions according to certain abnormal patterns. In step S102, abnormality identification is performed on each node action to obtain an identification result corresponding to each node action. The specific steps may include: Get a real-time screenshot of the operation interface in the current execution state; Based on real-time screenshots, abnormal identification of each node action is performed according to the identification strategy to determine the identification result corresponding to each node action; the identification strategy is determined based on predetermined abnormal patterns corresponding to multiple node actions, and the predetermined abnormal patterns are associated with the operation type of the corresponding node action.

[0059] In some embodiments provided in the present application, a real-time screenshot of the operation interface in the current execution state is obtained. Based on the real-time screenshot, the interface state changes can be captured instantly and anomalies can be quickly discovered, which is more intuitive and efficient than the traditional code-based or log-based detection method. According to the identification strategy determined by the predetermined abnormal pattern associated with the node action operation type, abnormal identification is performed on the above-mentioned node action to determine the corresponding identification result. Different node actions adopt customized identification strategies, which reduces unnecessary detection steps and reduces computational overhead. In the interactive process of multiple action combinations, the intelligent agent is used to automatically customize the abnormal pattern according to the characteristics of each node action, which can cope with various abnormal situations in complex business scenarios and ensure the stability of the automated task.

[0060] Specifically, the above-mentioned intelligent agent can adapt to real-time detection of interface changes and dynamically adjust the operation path. It can handle page structure change scenarios, such as element position offset and business rule mutations.

[0061] Exemplarily, there can be multiple types of the above-mentioned node actions, such as: click actions, input actions, drag actions, scrolling actions, judgment actions, process control actions, element interaction operations, hotkey operations, etc. The following describes the types and exception modes one by one.

[0062] Abnormal modes for click-type actions include interface elements not being found, clicks not being responsive, and the expected effect after clicking not occurring. Element not being found may be due to an update to the operating interface, resulting in the interface element not existing in the interface's DOM (Document Object Model), or it may be due to the interface element being covered by a pop-up window or floating layer. Clicking not being responsive may be due to a disabled state or unclickable attributes making the interface element uninteractive, or it may be due to incomplete loading of the interface element. The expected effect after clicking not occurring may be due to a problem where clicking a link did not respond, resulting in the page not jumping, clicking a button did not trigger a modal box, resulting in the pop-up not being displayed, or a checkbox did not switch to a selected state, resulting in a state not being updated.

[0063] Abnormal modes for input actions include the input box being unavailable and the input content not meeting expectations. The unavailable input box may be due to property settings that prohibit the input box from being used, or it may be in a read-only state with no input permission.

[0064] Drag anomalies include abnormalities in the start or end element, interruption of the drag process, and unexpected drag results. These abnormalities may be caused by some interface elements not supporting dragging, such as the start / end element not existing or being invisible. The termination of the drag process may be caused by page scrolling, resulting in a position shift and a drag distance exceeding the threshold. Unexpected drag results may include the dragged element not moving to the target or the action not being triggered after dragging.

[0065] Scrolling exceptions include: no scrolling, incorrect scroll position, and scroll timeout. A scrolling failure may be caused by a driver-page interaction failure, resulting in the scrolling operation not being triggered. An incorrect scroll position may be caused by not scrolling to the target element, scrolling farther than expected, causing the target element to be out of the viewport, or a scrolling operation stuck in an infinite loop, causing a loading exception where the scrolling continues to the bottom. A scrolling timeout may be caused by slow page loading, resulting in a scroll wait timeout.

[0066] Exceptions to judgment actions include: waiting timeout and conditional judgment failure. A waiting timeout occurs when an interface element fails to appear or refresh within the specified time. Conditional judgment failures can include the expected interface element not existing, for example, a drop-down menu that should automatically retract but remains in the pulled-down state, preventing the underlying interface element from being recognized. Alternatively, the state of an interface element may not match expectations, such as being non-interactive.

[0067] Exception modes for process control actions include: process interruption exceptions, such as interface elements not responding due to manual intervention or network interruption.

[0068] Exceptions to element interaction include: file upload failure, drop-down failure, and pop-up window processing failure. File upload failures may be caused by an error such as a non-existent file path or an upload timeout. Drop-down exceptions may be caused by the drop-down menu not expanding correctly, an empty drop-down menu, or the drop-down menu failing to perform the intended action (such as sorting or filtering the intended content). Pop-up window processing failures may include the inability to close the pop-up window or the pop-up window content displaying an incorrect message.

[0069] Abnormal patterns for hotkey operations include the page intercepting default keyboard events, such as disabling copy and paste for email or passwords, or abnormal character input, such as abnormal input switching.

[0070] In some embodiments provided herein, in step S103: generating a target action sequence starting with the target node action includes: Determine the response action for the predetermined abnormal pattern corresponding to the target node action; Generate target action sequence based on response actions.

[0071] In the embodiment provided by the present application, when an abnormality is identified in the target node action, the intelligent agent can match the corresponding response action from the preset response strategy library according to the predetermined abnormal pattern corresponding to the target node action. The above-mentioned response action is a repair or retry logic designed for a specific abnormal pattern. The intelligent agent takes the target node action as the starting point and organizes the matched response actions into an orderly execution sequence to form a target action sequence. The target action sequence is used to replace or supplement the problem part in the original global action sequence to ensure that the process can continue to execute after the abnormality is handled.

[0072] By generating this sequence of target actions, the system can dynamically adjust execution paths based on real-time exceptions, enhancing the flexibility and adaptability of automated processes. Centrally managing the mapping between exception patterns and response actions allows code maintainability by simply updating the response strategy library when business scenarios change, eliminating the need to modify the entire automated process.

[0073] For example, by recording the frequency of exceptions and the success rate of response actions, the exception handling strategy can be continuously optimized and the need for manual intervention can be gradually reduced.

[0074] Exemplarily, the above-mentioned application action settings can be based on the following methods: giving priority to lightweight repairs such as retrying and waiting, adding a confirmation mechanism for irreversible operations (such as submitting a form), and saving context data (screenshots, logs, state variables) at the time of the exception to facilitate backtracking and optimization of the intelligent agent.

[0075] The following describes the corresponding application actions for the above-mentioned abnormal modes.

[0076] If an interface element is not found, the response actions may include the following: retrying to locate the element (increasing the waiting time); checking whether there is a pop-up window / floating layer on the interface and closing it; triggering a page refresh and relocating; calling OCR (Optical Character Recognition) technology to identify the element position in the screenshot instead of DOM positioning.

[0077] For clicks that do not respond, the response actions may include the following: check whether the element is disabled, obtain administrator privileges, try to clear the disabled attribute; wait for the element to load. If the expected effect does not appear after clicking, the response actions may include the following: check whether the page URL has changed. If not, retry the click or switch the link; wait for the pop-up window / modal box to appear (set a timeout threshold); ③ verify whether the status of the interface element is updated; and re-execute the click process after rolling back the page.

[0078] If the input box is unavailable, the response actions may include the following: obtaining permission by logging in, or using administrator privileges to remove the disabled or read-only attribute of the input box; checking whether prerequisite operations (such as checking the agreement) need to be completed first; if the input box is hidden, try to scroll to the visible area before operating.

[0079] If the input content does not meet expectations, the response actions may include the following: verify whether the input format matches the regular expression; clear the original content in the input box and re-enter; check whether there is any automatic filling interference and clear the cache.

[0080] For abnormalities in the drag start or end elements, the response actions may include the following: handling element offset caused by page scrolling, scrolling to the target position before dragging, or dynamically adjusting the drag path to compensate for scroll offset.

[0081] Actions to address interruptions during a drag and drop process include: increasing the drag and drop timeout; dragging in segments (e.g., dragging to the middle point first and then to the end point); checking whether there are obstacles in the drag path (e.g., pop-ups) that could cause premature closing; using explicit waits to ensure elements are interactive; and recording the current drag and drop progress, resuming from the breakpoint after an interruption.

[0082] If the dragging result does not meet expectations, the response actions may include the following: verifying whether the target position coordinates are correct, such as comparing the pixel coordinates in the screenshot; checking whether additional events need to be triggered after the dragging; and re-executing the dragging after rolling back the page state.

[0083] For scrolling that has no effect or is in the wrong position, the response actions may include the following: click on a blank area of ​​the page to get the focus before scrolling; scroll in small increments multiple times instead of one-time scrolling; calculate the offset of the target element relative to the viewport; disable the infinite scrolling function; record the page height before scrolling and roll back in case of abnormality; use OCR to identify whether the page content after scrolling matches expectations.

[0084] For scrolling timeouts, the response actions can include the following: increasing the page loading timeout threshold; checking the network status and switching to a stable network.

[0085] For waiting timeouts, the response actions may include the following: extending the waiting time or switching networks; using event listening to check the status of interface elements; triggering a page refresh and then waiting again.

[0086] If the condition judgment fails, the response actions may include the following: re-obtaining the status of the interface element; comparing the interface screenshots to confirm whether the element actually exists; checking whether the business logic requires preconditions (such as login status).

[0087] For process interruption exceptions, the response actions may include the following: automatically retry the interrupted process, and the maximum number of retries can be set; send notifications to remind manual intervention, save the current process status, and resume from the breakpoint next time, and record interface screenshots and logs at the time of interruption to facilitate manual investigation.

[0088] For file upload failures, the response actions may include the following: verify the validity of the file path; check the file format / size limit, compress or convert the format; upload in chunks instead of uploading all at once.

[0089] For pull-down failures, the response actions may include the following: repeatedly trying to interact with the interface elements according to the predetermined number of retries; checking whether the options are dynamically loaded and waiting for data to be returned.

[0090] If pop-up window processing fails, the response actions may include the following: closing the browser tab and reopening the page; clicking on a blank page to trigger the pop-up window to automatically close.

[0091] When a page intercepts keyboard events, its response action can be to use a virtual keyboard plug-in to input special characters.

[0092] For abnormal character input, the response actions may include the following: restarting the input method; clearing the default value of the input box before entering again; verifying whether there is an input length limit and truncating the content.

[0093] According to the above embodiments and optional embodiments, the present application also provides an optional implementation method: Figure 2 A schematic diagram of an action sequence execution method according to an embodiment of the present application is shown. Figure 2 As shown in the figure, taking the user instruction "Order braised chicken with rice for less than 30 yuan, with fast speed and high score" as an example, the intelligent agent executes the processing flow in the takeaway application, extracts semantic constraints through the VLA model, determines that the dish type is braised chicken with rice, the price must be less than or equal to 30 yuan, and the sorting priority is speed over score.

[0094] According to the historical action sequence, the intelligent agent identifies five key node actions, including: Node 1: Sorting, Node 2: Store Selection, Node 3: Takeout Screening, Node 4: Ordering, and Node 5: Payment.

[0095] For node 1 sorting, the action is to sort by fastest speed and highest score. There are nodes that interact with pop-up and drop-down menus. The anomaly is that the sorting drop-down menu does not automatically retract, obscuring part of the store display. The response is to perform visual occlusion analysis or DOM node overlap detection on the menu retraction to determine whether a page element is visually obscured by other elements or overlapped by the layout. If an anomaly occurs, an action can be added to retract the menu by clicking on a blank screen.

[0096] For node 2, selecting a store, the action is to access the target store page for the Braised Chicken Rice dish. Abnormal patterns include the store being closed or the store information suddenly changing, such as the minimum delivery price suddenly changing to 40 yuan. The response can be based on the store's operating status, using icon recognition combined with OCR to read text prompts. For changes in store information, logical judgment can be made using DOM value extraction combined with pre-defined constraints.

[0097] For takeout food filtering at Node 3, the action is to select the "under 30 yuan" filter tag. Abnormal patterns include filtering an empty set, filtering condition drift, and filtering out items outside the acceptable price range. For an empty set anomaly, the response can include counting list items or performing visual inspection. For condition drift anomalies, OCR can be used to identify prices and perform value range verification.

[0098] For order placement at node 4, the action is to add the item to the shopping cart and then submit the order. Abnormal patterns include items suddenly sold out or items already in the shopping cart. The action can be to use the selectable button status, combined with visual inspection, to determine inventory anomalies, triggering a backtracking action and reselecting a store. To address the issue of items already in the shopping cart, the shopping cart list can be directly checked to see if it is empty.

[0099] For payments at Node 5, the action is to select the default or preferred payment method. Exceptions can include insufficient default payment balance or a payment interface loading timeout. Responses to payment failures can be identified through error text recognition or payment button status detection. Timeouts can be identified through a combination of a pre-set timer and loading animation recognition.

[0100] In the event of a real-time anomaly described above, the initially planned process is no longer applicable and requires a re-planning of the process, with alternative subsequent handling methods. Specifically, adaptively add steps to address the anomaly at each node. Only the affected nodes and subsequent steps are modified. For example, if node 1 is anomaly, insert a drop-down menu retraction action and continue the original process.

[0101] All of the above steps are executed using selectable actions within a predefined standard action space. This includes steps in the historical action sequence, which are also standardized actions in the form of action type + coordinates / content. Furthermore, these actions can utilize predictive boxes for GUI elements to increase reliability during operation.

[0102] This process significantly improves the efficiency of multi-step task processing in web automation scenarios, significantly enhancing the versatility and flexibility of intelligent agents. It adapts to a variety of page changes and dynamic content, demonstrates strong robustness and generalization, and is easily migrated to different web systems. Leveraging multimodal deep learning models, it can understand complex GUI information and achieve reliable human-machine interface process automation. Standardized operation output facilitates expansion and integration with other automation platforms, supporting subsequent secondary development and customized operations and maintenance.

[0103] Figure 3 A block diagram of an action sequence execution device according to an embodiment of the present application is shown. Figure 3As shown, corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application further provides an action sequence execution device, including: A generating module 301 is configured to generate a global action sequence based on semantic features representing user instructions, wherein the global action sequence includes a plurality of node actions having an execution order; The abnormality identification module 302 is used to identify abnormalities of each node action after executing each node action and obtain an identification result corresponding to each node action; The re-planning module 303 is used to generate a target action sequence starting from the target node action when the recognition result indicates that the target node action is abnormal; the target action sequence is used to update the global action sequence; The updating module 304 is configured to process the interface elements in the operation interface according to the updated global action sequence until the execution target of the user instruction is completed.

[0104] The functions of each module in each device in the embodiment of the present application can be referred to the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0105] Figure 4 FIG. 1 is a block diagram of an electronic device for implementing an embodiment of the present application. Figure 4 As shown, the electronic device includes: a memory 401 and a processor 402. The memory 401 stores a computer program that can be executed on the processor 402. When the processor 402 executes the computer program, the method of the above embodiment is implemented. The number of memory 401 and processor 402 can be one or more. In a specific implementation, the electronic device may also include a communication interface 403 for communicating with external devices and exchanging data.

[0106] In a specific implementation, if the memory 401, processor 402, and communication interface 403 are implemented independently, the memory 401, processor 402, and communication interface 403 can be connected to each other via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0107] Optionally, in a specific implementation, if the memory 401 , the processor 402 , and the communication interface 403 are integrated on a chip, the memory 401 , the processor 402 , and the communication interface 403 may communicate with each other through an internal interface.

[0108] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present application when the program is executed by a processor.

[0109] An embodiment of the present application provides a computer program product, including a computer program, which implements the method provided in the embodiment of the present application when executed by a processor.

[0110] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided in the embodiment of the present application.

[0111] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory. The input interface, the output interface, the processor and the memory are connected through an internal connection path. The processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0112] It should be understood that the processor described above may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor. It is worth noting that the processor may be a processor that supports the Advanced RISC Machines (ARM) architecture.

[0113] Furthermore, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache memory. By way of example and not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM) and direct memory bus random access memory (DR RAM).

[0114] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0115] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials, or characteristics described may be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless they are mutually inconsistent.

[0116] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. Throughout the description of this application, "plurality" means two or more, unless otherwise specifically defined.

[0117] Any process or method described in the flowchart or otherwise described herein can be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process. The scope of the preferred embodiments of the present application includes other implementations in which the functions may be performed in a different order than shown or discussed, including performing the functions substantially simultaneously or in reverse order depending on the functions involved.

[0118] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from and execute instructions on an instruction execution system, apparatus or device), or used in conjunction with such instruction execution systems, apparatuses or devices.

[0119] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0120] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the aforementioned integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium. The storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc.

[0121] The above are merely exemplary embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various modifications or substitutions within the technical scope described in this application, and such modifications or substitutions should be included within the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for executing an action sequence, characterized in that: include: Generate a global action sequence based on semantic features representing user instructions, wherein the global action sequence includes a plurality of node actions having an execution order; After executing each node action, performing abnormality identification on each node action to obtain an identification result corresponding to each node action; When the recognition result indicates that the target node action is abnormal, generating a target action sequence starting from the target node action; the target action sequence is used to update the global action sequence; According to the updated global action sequence, the interface elements in the operation interface are processed until the execution target of the user instruction is completed.

2. The method according to claim 1, characterized in that The generating of a global action sequence based on the semantic features representing the user instructions includes: Obtaining an initial screenshot of the operation interface; Determining an action execution strategy based on the initial screenshot, the semantic features, and a historical action sequence, wherein the historical action sequence is obtained based on historical node actions that complete historical instructions; According to the action execution strategy, a reference action is selected in a predetermined action space to generate the global action sequence.

3. The method according to claim 2, characterized in that The method further comprises: For predetermined type operations, extract multiple reference actions; The action space is generated according to the action types respectively corresponding to the multiple reference actions, the positioning of each interface element, and / or the reference filled content.

4. The method according to claim 3, characterized in that The method further comprises: Obtaining fusion features based on the visual features of the operation interface and the semantic features of the user instructions; Based on the fusion features, a region frame corresponding to each interface element is generated as the positioning, and the region frame is used to represent the interactive area of ​​the corresponding interface element.

5. The method according to claim 1, wherein The multiple node actions belong to process actions included in the global action sequence, and the method further includes: A node recognition model is used to determine a process action for changing an execution state from among multiple process actions as the multiple node actions. The node recognition model is obtained by training based on multiple historical action sequences with labeled historical nodes.

6. The method according to any one of claims 1 to 5, characterized in that The multiple node actions correspond to different abnormal modes respectively, and abnormality identification is performed on each node action to obtain an identification result corresponding to each node action, including: Obtain a real-time screenshot of the operation interface in the current execution state; Based on the real-time screenshot, abnormal identification of each node action is performed according to the identification strategy to determine the identification result corresponding to each node action; the identification strategy is determined based on the predetermined abnormal patterns corresponding to the multiple node actions, and the predetermined abnormal patterns are associated with the operation type of the corresponding node action.

7. The method according to claim 6, characterized in that The generating of a target action sequence starting from the target node action includes: Determining a response action for a predetermined abnormal mode corresponding to the target node action; The target action sequence is generated according to the response action.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

9. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images