Task automatic execution method, system, device and storage medium

Abstract

The embodiment of the application provides a task automatic execution method, system, device and storage medium, the method comprises the following steps: in response to receiving task inquiry information, determining the basic information related to the task inquiry information; acquiring execution information matched with the basic information, the execution information comprising an operation description sequence; the execution information is generated in advance by a cloud-side calling execution rule generation model based on the basic information; and performing operation on the to-be-operated client according to the operation description sequence to obtain a task execution result corresponding to the task inquiry information. By using the implementation mode, after receiving the task inquiry information, the operation process of the task is automatically completed throughout according to the execution information, and the abandonment rate of the user due to complicated operation steps is effectively reduced.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a method, system, device and storage medium for automatic task execution. Background Technology

[0002] In online shopping scenarios, it's becoming increasingly common for users to initiate pre-sales inquiries, look up online activity entry points, and provide operational guidance. The mainstream approach to handling these guidance inquiries is for the intelligent service system to return text instructions, operation path descriptions, or function screenshots to the user, allowing them to follow the steps outlined in the guidance.

[0003] This approach requires users to manually complete multiple triggering steps. This results in a long operation chain, leading to a high user abandonment rate. Furthermore, if the guidance information provided by the intelligent service system does not match the interface layout of the client to be operated, users may be unable to complete the operation according to the guidance information, thereby further reducing the user experience. Summary of the Invention

[0004] To overcome the problems existing in related technologies, embodiments of this application provide a method, system, device and storage medium for automatic task execution.

[0005] According to a first aspect of the embodiments of this application, a method for automatically executing tasks is provided, the method comprising: In response to receiving a task query message, determine the basic information related to the task query message; Obtain execution information matching the basic information, the execution information including an operation description sequence; the execution information is pre-generated by the cloud-based execution rule generation model based on the basic information; The operation is performed on the client to be operated according to the operation description sequence to obtain the task execution result corresponding to the task query information.

[0006] According to a second aspect of the embodiments of this application, a method for automatically executing tasks is provided, the method comprising: In response to receiving basic information, the execution rule generation model is invoked to generate execution information based on the basic information; the execution information includes an operation description sequence; the basic information is determined by the task execution end after receiving the task query information; The execution information is sent to the task execution terminal so that the task execution terminal performs operations on the client to be operated according to the operation description sequence, so as to obtain the task execution result corresponding to the task query information.

[0007] According to a third aspect of the embodiments of this application, an automatic task execution system is provided, the system including a task execution terminal and a cloud, wherein the cloud deploys an execution rule generation model. The task execution terminal is used to determine basic information related to the task query information after receiving the task query information; The cloud platform is used to invoke the execution rule generation model to generate execution information based on the basic information, and the execution information includes an operation description sequence. The task execution terminal is also used to obtain the execution information that matches the basic information, and to perform operations on the client to be operated according to the operation description sequence, so as to obtain the task execution result corresponding to the task query information.

[0008] According to a fourth aspect of the embodiments of this application, an automatic task execution apparatus is provided, the apparatus comprising: The determination module is used to determine basic information related to the task query information in response to receiving the task query information; The acquisition module is used to acquire execution information that matches the basic information, the execution information including an operation description sequence; the execution information is pre-generated by the cloud-based execution rule generation model based on the basic information. The operation module is used to perform operations on the client to be operated according to the operation description sequence, so as to obtain the task execution result corresponding to the task query information.

[0009] According to a fifth aspect of the embodiments of this application, an automatic task execution apparatus is provided, the apparatus comprising: The invocation module is used to invoke the execution rule generation model in response to receiving basic information to generate execution information based on the basic information; the execution information includes an operation description sequence; the basic information is determined by the task execution end after receiving the task query information; The sending module is used to send the execution information to the task execution terminal, so that the task execution terminal performs an operation on the client to be operated according to the operation description sequence, so as to obtain the task execution result corresponding to the task query information.

[0010] According to a sixth aspect of the embodiments of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method as described in the first or second aspect.

[0011] According to a seventh aspect of the embodiments of this application, a computer-readable storage medium is provided having a computer program stored thereon, the program being executed by a processor to implement the method as described in the first or second aspect.

[0012] According to an eighth aspect of the embodiments of this application, a computer program product is provided, including a computer program that is executed by a processor to implement the method of the first aspect or the second aspect.

[0013] The technical solutions provided in this application embodiment may include the following beneficial effects: Upon receiving the task query, the task execution end obtains execution information containing a sequence of operation descriptions and automatically executes the corresponding operations on the client side according to this information, thus directly obtaining the task execution result. It is evident that in this process, the user does not need to manually follow the steps; the entire task operation process is completed automatically, effectively reducing user abandonment rates due to cumbersome operation steps. Furthermore, the execution information is generated by the cloud based on basic information related to the task query, and this basic information is related to the client side. In other words, the execution information matches the client side, thus avoiding operation failures caused by mismatches between the execution information and the client's interface layout. This improves the processing efficiency and user experience of operation guidance-type consultations. Moreover, in this technical solution, the cloud-based model generation reduces the computational overhead on the task execution end, shortening the time from user initiation of a query to obtaining execution information, thereby reducing user waiting time.

[0014] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly described below. It should be understood that those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0016] Figure 1 A schematic diagram of an exemplary task automatic execution system provided in this application embodiment; Figure 2A An exemplary method flowchart of the task automatic execution method provided in the embodiments of this application; Figure 2B A schematic diagram of another exemplary method flow for the automatic task execution method provided in the embodiments of this application; Figure 3 An exemplary signaling interaction diagram of the task automatic execution method provided in the embodiments of this application; Figure 4A Exemplary composition diagram of the task automatic execution device provided in the embodiments of this application Figure 1 ; Figure 4B A second exemplary schematic diagram of the task automatic execution device provided in the embodiments of this application; Figure 5 This is an exemplary structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings.

[0018] The terminology used in the following embodiments of this application is for the purpose of describing specific embodiments and is not intended to limit the technical solutions of this application. As used in the specification and appended claims of this application, the singular expressions "a," "an," "the," "the," "the," and "this" are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that although the terms first, second, etc., may be used in the following embodiments to describe a class of objects, the objects are not limited to these terms. These terms are used to distinguish specific implementations of that class of objects.

[0019] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0020] The following describes the implementation scenarios of this application.

[0021] This application relates to online dialogue service technology, primarily to the technology for processing user operation guidance inquiries. User operation guidance inquiries refer to responses to queries that help users perform tasks, such as inquiries about coupon redemption methods, promotional activity entry points, and coupon usage rules. For operation guidance inquiries, the intelligent service system typically queries a pre-set knowledge base for matching text descriptions, operation path descriptions, or function screenshots, such as an operation path description like "Click 'My'—'Benefits Center'—'Coupons' to enter the redemption page," allowing users to follow the steps indicated in the guidance information.

[0022] However, this approach has several drawbacks. First, it requires users to manually complete multiple triggering steps, resulting in a long operation chain that users must memorize, leading to a high abandonment rate. Second, the version number of the client to be operated (such as the client of an e-commerce platform) is frequently updated, causing frequent changes in the page layout, control hierarchy, and style of the client. Meanwhile, the guidance information in the knowledge base is usually not updated frequently. If the guidance information provided by the intelligent service system does not correspond to the actual interface of the client to be operated, users may be unable to complete the operation according to the guidance information, further reducing the user experience.

[0023] In view of this, the technical solution provided in this application, upon receiving task query information, obtains execution information containing a sequence of operation descriptions, and automatically executes the operations corresponding to each operation description sequentially on the client to be operated according to the execution information, thereby directly obtaining the task execution result. In this technical solution, the execution information is generated based on the version number of the client to be operated related to the task query information; that is, the execution information matches the version number of the client to be operated, which helps improve the processing efficiency of operation guidance-type consultations. Therefore, by adopting this technical solution, users do not need to manually follow the steps to perform operations; the entire task operation process is completed automatically, thereby effectively reducing the user abandonment rate due to cumbersome operation steps.

[0024] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0025] The automatic task execution method provided in this application embodiment can be applied to an automatic task execution system, such as... Figure 1 As shown, an exemplary task automation system may include a task execution terminal 10 and a cloud terminal 20. The task execution terminal 10 may be deployed in an electronic device such as a mobile terminal, smart bracelet, smartwatch, computer, game console, or personal computer (PC). The cloud terminal 20 may be deployed in an electronic device such as a computer, game console, or personal computer (PC), or in a cloud computer, distributed cluster, etc. A network connection exists between the task execution terminal 10 and the cloud terminal 20.

[0026] The task execution terminal 10 is configured to interact with the user and trigger the client to perform user operation tasks. The task execution terminal 10 can receive task query information input by the user and collect basic information related to the task query information. This basic information is metadata used to determine the execution information, and may include, for example, the version number of the client to be operated, the intent type corresponding to the task query information, and a promotional activity identifier. The task execution terminal 10 can parse the execution information to obtain the operation description sequence and the corresponding page view, as well as the semantic representation of the interface elements corresponding to each operation description. Based on the parsed information, it can automatically execute the operation task corresponding to the query information. The client to be operated can be the object to which the operation task is directed. In some embodiments, the client to be operated can be implemented as an independent application (APP). An APP refers to a software program that can run directly on the operating system without relying on other applications. In some embodiments, the client to be operated can be implemented as a lightweight program mounted in other APPs, also known as a "mini-program." For example, the client to be operated can be an APP of a shopping platform.

[0027] In some embodiments, the task execution terminal 10 deploys a dialogue client, an edge execution engine, a memory, and a protocol layer. The dialogue client can be coupled to the client to be operated, providing an interaction point between the user and the edge execution engine or protocol layer, and displaying task execution results to the user, by setting the access entry point in the client to be operated. For example, the dialogue client may be an instant messaging (IM) client. The edge execution engine is the controller that triggers the automatic execution of the task and can be deployed in the dialogue server. For example, the edge execution engine can be coupled to the intelligent dialogue module of the dialogue server. The edge execution engine can maintain a control strategy for the automatic execution of the task. The control strategy may include a strategy for the edge execution engine to collect basic information and read execution information before obtaining execution information, and a strategy for the edge execution engine to automatically execute the task based on the execution information after obtaining execution information, such as parsing to obtain the view sequence and determining the interface elements to be triggered in the page view. The edge execution engine also supports communication with the cloud, thereby supporting sending task query information to the cloud and receiving execution information from the cloud. The memory is used to cache local data and information on the task execution end 10. For example, it can cache some execution information of operation tasks. This allows the edge execution engine to look up execution information locally, which helps improve the speed of reading execution information and the efficiency of automatic task execution. In some embodiments, the execution information can conform to the data transmission protocol between the edge execution engine and the cloud, such as the Robotic Process Automation (RPA) protocol. Based on this, the protocol layer is used to parse the execution information and obtain automatic execution permissions from the user through the dialogue client.

[0028] The RPA protocol is an execution specification for the intermediate representation (IR) layer in the field of automated interaction, located between large model inference and edge execution. RPA protocol information, i.e., execution information, is a parsable and executable operational flow specification for the edge execution engine, obtained by transforming the abstract intent generated by the execution rule generation model. Therefore, the technical solution of this application relies on the RPA protocol to achieve mutual collaboration yet decoupling between cloud-based decision-making and edge execution, enabling flexible adaptation to a variety of task execution scenarios.

[0029] The task execution terminal 10 can be carried out on an electronic device equipped with a display. The display is configured to show the user interface (UI) of the client to be operated corresponding to the task execution terminal, and to provide an interactive interface for the dialogue client deployed in the task execution terminal. The UI involved in this application embodiment is configured as a medium interface for delivery personnel to operate and interact with the application. As an interaction interface with the user, the UI can convert the computer language of the application into a form that the user can accept and recognize, including displayed images, text, buttons, etc. A common form of UI is a graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of an electronic device. The control can include visual interface elements such as icons, buttons, menus, tabs, and text boxes. The control can be implemented as a visual trigger interface to advance the operation task in the operation task chain. When the control receives a corresponding trigger operation from the user, the client page of one link in the operation task chain is updated to the client page of the next link, thereby causing the client to be operated to automatically execute the operation task in response to the operation description sequence. For example, when the "My" control on the current page of the client being operated responds to a virtual click command, the current page is updated to the user's personal homepage.

[0030] Cloud 20 is the management engine of the task automation system. It can be configured to manage and maintain the page navigation topology of clients to be operated for each version number, as well as activity information for various promotional activities. Based on this, Cloud 20 can support intent recognition from basic information from task execution terminal 10, customize operation description sequences based on intent and page navigation topology, and encapsulate information containing operation description sequences to obtain execution information. For example, this could be RPA protocol encapsulation. In some embodiments, execution rule generation models can be deployed in Cloud 20 to perform intent recognition, operation description sequence generation, RPA protocol encapsulation, and other processing. For example, the execution rule generation model can be a Large Language Model (LLM). LLMs can include Generative Pre-trained Transformer (GPT) series models or other large language models. Other large language models can include, for example, Autoencoder Based Models (ABM), Sequence-to-Sequence Models (SSM), Transformer-Based Models (TBM), etc.

[0031] As can be seen, this technical solution deploys the large-scale model processing logic in the cloud, that is, the processing procedures that consume a lot of computing resources and time, in the cloud, while deploying high-frequency, low-latency execution on the edge. In this way, by using a collaborative approach between edge and cloud testing, the tasks corresponding to user-initiated operation guidance inquiries can be automatically executed.

[0032] In some embodiments, the cloud 20 can maintain edge storage nodes and a cache gateway. The edge storage nodes are used to store execution information of some operation tasks, so that if the task execution terminal 10 does not find matching execution information in the local cache, it can read the matching execution information from the edge storage nodes. This eliminates the need to call the execution rule generation model to repeatedly generate execution information, which helps to improve the response speed to task query information and enhance the user experience. For example, the edge storage nodes can be storage nodes in a distributed storage system, a Content Delivery Network (CDN), or an Object Storage Service (OSS) system.

[0033] The caching gateway is used to determine the nearest edge storage node to the task execution terminal 10 upon receiving a query request from the task execution terminal 10, and to search for matching execution information from the edge storage node. The caching gateway is also used to store execution information from the execution rule generation model in the edge storage node.

[0034] It should be understood that the above description of the automatic task execution system is merely illustrative and does not constitute a limitation on the automatic task execution system in the embodiments of this application. In actual implementation scenarios, the automatic task execution system may include more or fewer functional modules. The embodiments of this application do not limit this.

[0035] Please refer to Figure 2A , Figure 2A An exemplary method for automating task execution is shown, which can be implemented by... Figure 1 The task execution terminal is shown in the diagram. This automated task execution method may include the following steps: In step S101, in response to receiving task query information, basic information related to the task query information is determined.

[0036] In step S102, execution information matching the basic information is obtained, and the execution information includes an operation description sequence.

[0037] In step S103, the operation is performed on the client to be operated according to the operation description sequence to obtain the task execution result corresponding to the task query information.

[0038] The task query information can be implemented as natural language text or instructions expressing task requirements. For example, the task query information can be text describing the requirements of an operation task, such as queries like "Claim a coupon" or "Entry point to participate in a promotional activity." As can be seen from the description of the foregoing embodiments, the task execution terminal can receive task query information input by the user through a dialog box of a dialogue client or a voice input interface.

[0039] Since the basic information is metadata used to determine execution information, and in order to obtain accurate execution information, the basic information should include a multi-dimensional contextual data set that uniquely represents the applicable scenario of the execution plan. In some embodiments, the basic information includes at least the version number of the client to be operated. In some embodiments, the basic information may also include, but is not limited to, the identifier of the currently ongoing promotional activity, the user identifier, and the user device model.

[0040] Based on this, since the execution information is pre-generated by the cloud-based execution rule generation model based on the aforementioned basic information, the execution information matches the version number of the client to be operated on. Therefore, the execution information and the page layout of the client to be operated on can be accurately matched, thus avoiding the problem of operation failure caused by mismatch between the execution information and the interface layout of the client to be operated on. This is beneficial to improving the processing efficiency and user experience of operation guidance-type consultations. Furthermore, in this technical solution, generation by the cloud-based model helps reduce the computational overhead on the task execution side and shortens the time from when the user initiates an inquiry to when they obtain the execution information, thereby reducing the user's waiting time.

[0041] Furthermore, the execution information includes a sequence of operation descriptions. This sequence can refer to an ordered set of multiple atomic operations arranged in a predetermined execution order. Each atomic operation can be an action unit that the client-side execution engine can directly recognize and execute, such as a tap, scroll, or text input. After obtaining the execution information, the task execution end can automatically execute the operations corresponding to each operation description on the client to be operated, thereby directly obtaining the task execution result. Therefore, in this process, users do not need to manually follow the steps; the entire task operation process is completed automatically, effectively reducing user abandonment rates due to cumbersome operation steps.

[0042] In some embodiments, after receiving the task query information, the dialogue client can collect basic information from the runtime environment of the client to be operated, and then transmit both the task query information and the basic information to the edge execution engine. In other embodiments, after receiving the task query information, the dialogue client transmits the task query information to the edge execution engine. The edge execution engine collects basic information from the runtime environment of the client to be operated. For example, the basic information can be collected by reading the version number and user identifier of the client to be operated through an Application Programming Interface (API), and retrieving the identifier of the currently ongoing promotional activity from local storage.

[0043] In one alternative implementation, the semantics of the task query are somewhat ambiguous, and the specific task object is not explicitly specified. The task execution end can supplement the basic information based on the current context of the client to be operated on. For example, if the task query is "claim coupon," and the client to be operated on is currently displaying the details page of a certain product, then the task execution end can automatically use the product identifier or product category identifier as the basic information to improve the accuracy of the execution information.

[0044] Furthermore, after obtaining the basic information, a query key can be generated based on the task query information and the basic information. This query key serves as the basis for finding execution information that matches the basic information. For example, the query key can be a composite key, consisting of a hash value of the intent type, the version number of the client to be operated on, and an activity identifier. The intent type can be the category of the operation intent represented by the task query information. For instance, the query key could be: hash("Get Coupon")_v10.2.0_2026 Year-End Promotion), where "Get Coupon" refers to the intent type, v10.2.0 refers to the version number, and 2026 Year-End Promotion refers to the activity identifier.

[0045] In some embodiments, the task execution terminal can search for execution information matching the basic information from local storage. If the execution information is found, it reads the execution information. If the execution information is not found, it sends a search request carrying the basic information to the cloud and receives the execution information from the cloud. The execution information is searched or generated by the cloud.

[0046] Local storage can be a persistent or semi-persistent data storage medium deployed on the device where the edge execution engine resides. In some embodiments, local storage can be a sandboxed file system directory of the edge execution engine; in other embodiments, local storage can be implemented as a lightweight key-value database (such as SQLite or SharedPreferences) built into the edge execution engine.

[0047] It should be noted that the edge execution engine can receive multiple task query requests through different dialog clients. Any two task query requests can request the same or different operation tasks. Furthermore, if the execution information corresponding to any of the task query requests is not stored in the local storage, the edge execution engine can request the cloud to generate the corresponding execution information. After receiving the execution information from the cloud, the engine caches the received execution information locally. Based on this, the execution information stored in the local storage is all execution information that has been obtained before step S101. In this way, after determining the basic information, the corresponding execution information is first searched for in the local storage. If found, it is read directly, which not only improves processing efficiency but also helps avoid repeated requests to the cloud, thus reducing network bandwidth usage and cloud computing overhead.

[0048] For example, the edge execution engine can use the aforementioned query key as an index to traverse the index keys in local storage. After finding an index key that matches the query key, the execution information corresponding to that index key is used as the execution information matching the basic information. In one implementation, to manage the storage space of local storage, a storage expiration period (Time To Live, TTL) can be set for the execution information in local storage. In this embodiment, after finding the execution information corresponding to the query key, the edge execution engine can check whether the storage timestamp of the execution information has exceeded the expiration period. If it has not exceeded the expiration period, the execution information can be read. If it has exceeded the expiration period, the search result is determined as no execution information was found. The storage expiration period can be flexibly set according to the actual implementation scenario. If the current period is during a large-scale promotional activity with a relatively short duration, the storage expiration period can be set relatively short, for example, 7 days. If the current period is in a regular implementation phase, the storage expiration period can be set relatively long, for example, 30 days.

[0049] By adopting this implementation method and introducing a cache expiration mechanism, it is beneficial to manage the local storage space reasonably and avoid the misuse of historical planning information due to version updates of the client to be operated or dynamic changes in promotional activities.

[0050] A search request can be implemented as a network request message, used to initiate a request to the cloud to query the execution information corresponding to specified basic information. This network request message may carry additional information from the basic information, such as the version number of the client to be operated, the promotional activity identifier, the intent type, and the user identifier.

[0051] In some embodiments, the task execution terminal can receive the execution information from the caching gateway. In this embodiment, the execution information is returned to the task execution terminal after the cloud calls the caching gateway to find the execution information from the edge storage node. In other embodiments, if the caching gateway does not find the execution information from the edge storage node, the cloud calls the execution rule generation model to generate the execution information based on the basic information. In this embodiment, the task execution terminal can receive the execution information from the execution rule generation model. The specific implementation process of the cloud, the specific implementation methods of the caching gateway and the edge storage node are detailed in the following embodiments and will not be repeated here.

[0052] In some embodiments, after receiving the execution information returned from the cloud, the edge execution engine not only automatically executes the task according to the execution information, but also writes the planning information into local storage and sets the timestamp and validity period of the execution information so that when other users initiate requests for the same operation task, they can directly find the execution information from local storage.

[0053] As can be seen, by adopting this implementation method, execution information is searched in a hierarchical manner, prioritizing the search for execution information in local storage and then searching for execution information in edge storage nodes. This ensures that execution information in most high-frequency scenarios can be found in either local storage or edge storage nodes, allowing users to experience zero-latency response. Only when the execution information cannot be found in edge storage nodes is the execution rule generation model called to generate execution information, which reduces the frequency of calling the execution rule generation model, thereby reducing network bandwidth usage and cloud computing overhead.

[0054] Since automatically operating the client according to the execution information may lead to a lack of control and trust for the user, the task execution terminal can display automatic execution authorization information after obtaining the execution information. For example, the authorization information includes the operation description sequence, thereby informing the user of the automatically executed operation. Furthermore, if an instruction to grant permissions is received, the task execution terminal can perform the operation on the client according to the operation description sequence.

[0055] Combination Figure 1The illustrated embodiment describes a scenario where the task execution end can display automatic execution authorization information in a dialog box of the chat client, thereby showing the user this authorization information. For example, the task execution end can present the automatic execution authorization information in the IM client's dialog box in the form of a confirmation card. The central area of ​​the confirmation card can display the purpose of the automatic task, the steps to be performed, the scope of page navigation involved, and the estimated operation time, such as "Automatically claim promotional discount coupons for you," with the steps including "'My'—'Benefits Center'—'Coupons'," and an estimated operation time of 1 second. The confirmation card can include a confirm button and a refuse button. The confirm button is set as the entry point for granting permission, and the refuse button is set as the entry point for denying authorization. In response to receiving a click on the confirm button, an instruction to grant permission is transmitted to the task execution end; in response to receiving a click on the refuse button, an instruction to deny authorization is transmitted to the task execution end.

[0056] In some embodiments, the automatic execution authorization information may include the risk level of the current operation task. This risk level can be determined based on the operation type of each operation description. For example, if the operation description sequence includes operations involving changes to account assets, such as "confirm order" or "redeem coupon," it is marked as high risk and accompanied by corresponding warning information. If the operation description sequence only includes page browsing and information query operations, it is marked as low risk.

[0057] In some embodiments, the automatic execution authorization information may also include an "operation preview" area, which can display the operation flowchart or operation step list of the current operation task, so that users can intuitively understand the process of the automated operation through the content of the preview area, thereby better assisting users in making authorization decisions.

[0058] As can be seen, this embodiment, by setting up an authorization confirmation mechanism, enables users to fully understand the content, risks, and other information of automatically executed operations, thereby giving users full control and the right to know, which is conducive to improving the security of interaction and users' trust in the system.

[0059] Furthermore, after being granted automatic execution permission, the client-side execution engine can construct a sequence of page views corresponding to each operation description in the operation description sequence, and determine the page view in the page view sequence that matches the current page of the client to be operated, as the starting operation page view. Then, it can start from the starting operation page view and execute the corresponding operations sequentially according to the order of the page view sequence.

[0060] In this sequence, each page view corresponds one-to-one with a specific operation description in the operation description sequence. Each page view represents a UI interface that performs a corresponding atomic operation. For example, the atomic operation for the "Coupon Redemption Page" is a click operation, so the "Coupon Redemption Page" is the page view corresponding to the click operation. The current page represents the UI interface that the client-side execution engine actually displays on the screen after parsing the execution information and initiating the virtual operation. The client-side execution engine can determine the current page by obtaining the view tree information of the current page.

[0061] For example, during the stage where the cloud-based execution rule generation model generates execution information, it can annotate each atomic operation with a corresponding view page identifier. The view page identifier can be the page's route name or the hash value of the page's view tree root node. For example, the page's route name could be "home_page" or "coupon_center_page". Based on this, after parsing the execution information, the client-side execution engine can determine the atomic operations corresponding to each operation description and their corresponding view page identifiers. Furthermore, it can construct a page view sequence according to the order of the operation description sequence.

[0062] This embodiment constructs a page view sequence and matches it with the current page, enabling the client-side execution engine to start execution from the middle operation step of the operation description sequence. This helps to avoid meaningless page jump operations and shortens the total task execution time.

[0063] The client-side execution engine can identify page views in the page view sequence that are identical to the current page. If a page view in the page view sequence is identical to the current page, it is designated as the starting operation page view. If no page view in the page view sequence is identical to the current page, the first page view in the page view sequence is designated as the starting operation page view. The starting operation page view can locate the position where the task begins execution. If a page view in the page view sequence is identical to the current page, the page currently loaded by the client to be operated on can be determined, which is the trigger page for continuing the operation task. There is no need to trigger a page jump on the client to be operated on. The current page can be triggered according to the matching atomic operation to start automatically executing the operation task. For example, if the page view sequence corresponds to the page sequence including 'My' — 'Benefits Center' — 'Coupons', and the current page is the 'Benefits Center' page, the client-side execution engine can perform the corresponding virtual operation on the 'Benefits Center' page to enter the 'Coupons' page. If the current page does not match any of the page views in the page view sequence, it can be determined that the page currently loaded by the client is unrelated to the operation description sequence. In this case, the client-side execution engine operates on the current page to jump to the first page in the page view sequence, and then automatically executes the task starting from the first atomic operation corresponding to the operation description sequence. For example, if the page view sequence includes 'My' — 'Benefits Center' — 'Coupons', and the current page is the 'Payment' page, the client-side execution engine can trigger a jump from the 'Payment' page to the 'My' page, and then automatically execute the operation task from the 'My' page according to the operation description sequence.

[0064] In some embodiments, the determination that a page view is the same as the current page can be based on an exact match of page identifiers. For example, the client-side execution engine can match the route name of the current page with the preset route names of each page view in the page view sequence, and determine the page view whose preset route name matches the route name of the current page as the same page view as the current page. For instance, if the route name of the current page is MainHomePage, the page view with the preset route name MainHomePage will be considered the same page view as the current page.

[0065] In some embodiments, the similarity between a page view and the current page can be determined by matching page features in each page view. If a page view's page features match those of the current page, then that page view is determined to be the same as the current page. For example, page features may include the text in the top title bar, the selected state of the bottom navigation bar, and specific function buttons on the page.

[0066] It should be noted that if multiple page views in the page view sequence match the current page, the client-side execution engine can select the page view closest to the last atomic operation as the starting operation page view, according to their order in the page view sequence. This helps to minimize the number of operation steps, thereby improving execution efficiency.

[0067] In some embodiments, the execution information also includes semantic representations of the interface elements corresponding to each operation description. Based on this, after determining the starting operation page view, for the starting operation page view and any subsequent page view, the client-side execution engine can determine the interface elements in the page view according to the semantic representations corresponding to the page view, and perform matching virtual operations on the interface elements according to the operation descriptions corresponding to the page view.

[0068] The semantic representation of an interface element describes its semantic features. This semantic representation may include descriptive information across one or more semantic dimensions, but may not include the element's identifier or screen coordinates. For example, the semantic representation may include at least one of the following: the element's displayed text (e.g., "Claim Now"), the element's hierarchical position relative to its parent container or sibling elements, and the element's control type (e.g., "Button," "TextView"). In one embodiment, the semantic representation may be carried in the form of a semantic selector within RPA protocol instructions, forming execution information along with an operation description sequence.

[0069] In some embodiments, the view tree of the page view may contain attribute data of the visible nodes in the page view, and each visible node may correspond to a visual interface element of the page view. The semantic representation of any interface element may contain one or more semantic feature key-value pairs, such as {"text": "Get it now", "type": "Button"}. For example, after extracting the corresponding semantic representation of the page view, the client-side execution engine may traverse the attribute data of each node in the view tree of the page view, match the attribute data with the features in the semantic representation, and determine the interface element indicated by the node as the interface element to be operated when the attribute data of the node satisfies the semantic features in the semantic representation. In some embodiments, the client-side execution engine may use regular expressions or fuzzy matching to select the interface element to be operated. For example, the semantic representation {"text": ".*Get it now.*"} may indicate that any element whose text content contains the word "Get it now" can be matched. Therefore, interface elements whose attribute data contains "Get it now", "Get it immediately", or "Click to get it" can all be matched.

[0070] Virtual operations are commands triggered by simulated physical user actions, acting as virtual triggers on corresponding UI elements. Each operation description can include an operation type and operation parameters. The operation type can define corresponding virtual operations, including but not limited to: click, long press, swipe / scroll, text input, and back. The operation parameters define the execution amount of the virtual operation. For example, for a virtual operation of type "click," the client-side execution engine can call the click event injection interface on the located UI element, and then simulate a complete press and release operation. Similarly, for a virtual operation of type "swipe," the client-side execution engine can perform a swipe operation within the area where the UI element is located, based on the start and end coordinate offsets in the operation parameters.

[0071] It should be noted that if a UI element is a dynamic UI element whose state is periodically updated, the element cannot be manipulated during the state update process. Only after the UI element's state update is complete can it respond to virtual operations and perform subsequent processing. Therefore, before executing a virtual operation, the client-side execution engine can check the state of the UI element, such as whether it is clickable, visible, and enabled. If the UI element is currently unclickable or invisible, the client-side execution engine can wait for a predetermined time (e.g., 500 milliseconds) before relocating the UI element and executing the virtual operation. This effectively avoids the situation where virtual operations are performed on currently non-interactive UI elements, leading to execution failure.

[0072] By adopting this implementation method and describing interface elements through semantic representation, the client-side execution engine can accurately identify and locate interface elements without relying on fixed control identifiers or coordinates. This improves the tolerance for changes in page layout. Even if the page layout is slightly adjusted, such as changing button colors, adjusting spacing, or changing ad positions, the interface elements can still be accurately located and virtual operations can be performed, thus improving the positioning accuracy and execution success rate of automated operations.

[0073] In some embodiments, in response to a failed operation on the interface element, the client-side execution engine does not directly terminate the automatic task execution process. Instead, it detects the total number of repeated operations. If the total number is less than a preset threshold, the virtual operation is repeated on the interface element, and the total number is incremented by 1. If the total number equals the preset threshold, the process is transferred to a human agent. For example, the preset threshold can be 3, but this application is not limited to this.

[0074] The failure to operate on the interface elements may include situations such as the inability to locate the interface element, the interface element being in a non-interactive state, an error code being returned after the virtual operation is executed, or the expected page changes not occurring after the virtual operation is executed. Since operation failures may be caused by occasional reasons such as incomplete interface animations or response delays, a retry mechanism can be set up with a preset threshold number of retries. This provides a limited number of fault-tolerant recovery functions for automated operation failures. After the preset threshold is reached, retries are terminated promptly and the user is transferred to a human agent. This helps improve the success rate of automated execution while avoiding indefinite waiting for users, thus optimizing the user experience.

[0075] Please refer to Figure 2B , Figure 2B An exemplary method for automating task execution is shown, which can be implemented by... Figure 1 The cloud-based execution is illustrated in the diagram. The automated execution method for this task may include the following steps: In step S201, in response to receiving basic information, the execution rule generation model is invoked to generate execution information based on the basic information.

[0076] The basic information is determined by the task execution end after receiving the task query information. The execution information includes an operation description sequence.

[0077] For an exemplary implementation of the basic information and operation description sequence, please refer to [reference needed]. Figure 2A The relevant descriptions in the embodiments of this application will not be repeated here.

[0078] In step S202, the execution information is sent to the task execution terminal.

[0079] Upon receiving the execution information, the task execution terminal performs operations on the client to be operated on according to the operation description sequence to obtain the task execution result corresponding to the task query information. An exemplary implementation of the task execution terminal automatically executing tasks according to the execution information can be found in [reference needed]. Figure 2A The relevant descriptions in the embodiments of this application will not be repeated here.

[0080] By deploying the process of generating execution information in the cloud, this technical solution decouples cloud-based intelligent decision-making from local execution on the edge. On the other hand, it fully utilizes the abundant computing resources in the cloud for large-scale inference, allowing edge devices to undertake only lightweight parsing and execution tasks, effectively reducing edge resource consumption and version maintenance costs.

[0081] As described in the foregoing embodiments, the cloud receives basic information through a lookup request. This lookup request is sent by the task execution end when it cannot find the execution information in its local storage. It is a trigger message used to request the search for execution information from the edge storage node. Based on this, before calling the execution rule generation model to generate execution information based on the basic information, the cloud can call the cache gateway to search for the execution information from the edge storage node based on the basic information carried in the lookup request. If the execution information is found in the edge storage node, it is sent to the task execution end. If the execution information is not found in the edge storage node, the execution rule generation model is called to generate execution information based on the basic information.

[0082] For example, a lookup request can be implemented as a Hypertext Transfer Protocol (HTTP) GET request, in which the Uniform Resource Locator (URL) of the lookup request can carry fields of various dimensions of basic information, or in which the header field of the lookup request carries an encoded query key.

[0083] In some embodiments, prior to certain promotional activities, the cloud can invoke an execution rule generation model to pre-generate execution information that combines high-frequency intents with mainstream client versions to be operated, and pre-write this execution information into each edge storage node.

[0084] A caching gateway can be implemented as a service node deployed at the edge of the cloud network. It performs fast key-value lookups in edge storage nodes based on lookup requests from the task execution end, and stores execution information from the execution rule generation model in edge storage nodes. As distributed storage units, edge storage nodes are geographically close to the task execution end, providing low-latency information retrieval services. Therefore, execution information is preferentially retrieved from edge storage nodes. Since the lookup process does not involve large model inference, the response speed is fast, improving user experience.

[0085] As described in the foregoing embodiments, the execution rule generation model can be a large language model. Based on this, the cloud calls the execution rule generation model to generate execution information based on the basic information. Specifically, this can be achieved by identifying the intent type and activity identifier corresponding to the task query information, as well as the version number of the proxy operation client; obtaining the page navigation topology map corresponding to the activity identifier under the version number from a preset page navigation topology map library; then, generating the operation description sequence according to the intent type and the page navigation topology map; determining the semantic representation of the interface elements corresponding to each operation description; and encapsulating the operation description sequence and the semantic representation of the interface elements corresponding to each operation description according to a preset format to obtain the execution information.

[0086] Intent type refers to the type of task a user expects to complete, such as coupon redemption, event check-in, or points redemption. After obtaining basic information, the execution rule generation model can perform natural language understanding on the task query information within that information to infer the intent type that matches the query information among various predefined intent types. Activity identifiers are used to identify promotional activities, thereby distinguishing the interface layout and entry point location during different promotional periods, such as the 2024 annual holiday promotion and brand member days.

[0087] The page navigation topology library can be the operation navigation paths for different operation types of tasks corresponding to various promotional activities under different client versions. The page navigation topology graphs in the library can be generated by the online shopping server, and any page navigation topology graph can be indexed by the client version number and promotional activity identifier. In some embodiments, the page navigation topology graph can be implemented as a tree structure or a Directed Acyclic Graph (DAG) structure, representing the page hierarchy and jump path of the corresponding operation task. Taking a DAG implementation as an example, each node in the page navigation topology graph can represent a page in the client to be operated, as well as the semantic features of the interface elements on the corresponding page, and each directed edge can represent the operation behavior of jumping from one page to another. For example, pages in the client to be operated on include the homepage, the main event page, the coupon list page, and the coupon redemption success page. Each node (page) in the page navigation topology diagram is also labeled with the semantic description of the interface elements on the page, such as the page title, the list of core functional controls contained on the page, and the depth of the page in the navigation hierarchy. The operation described by any directed edge is, for example, "Click the 'coupon' icon in the bottom navigation bar to jump to the coupon list page".

[0088] In some embodiments, after determining the page navigation topology based on the client version number and promotion activity identifier in the basic information, the execution rule generation model can use the intent type as the target node and the root node in the page navigation topology as the starting node. It can use a graph search algorithm to find the shortest path from the starting node to the target node. Then, each directed edge in the shortest path is converted into a corresponding atomic operation. Subsequently, the execution rule generation model converts each atomic operation into an operation description sequence.

[0089] For example, if the intent type is to claim a coupon, and the root node in the page navigation topology is the homepage, then we determine the homepage as the starting node and "claim coupon" as the target node. We find the shortest path, which can be implemented as "homepage → main event page → coupon claim page (click the 'discount coupon' area)". After converting the atomic operations between "homepage → main event page" and between "main event page → coupon claim page" into operation descriptions, we can obtain, for example, the operation description sequence: [Jump to the main event page, enter the coupon claim page].

[0090] In some embodiments, during the process of converting each directed edge into a corresponding atomic operation, the execution rule generation model can extract the semantic features of the interface element corresponding to the operation from the node's annotation information. For example, for the operation of "clicking the 'Activity Main Venue Page' entry", the execution rule generation model can extract the semantic features of the entry control corresponding to the node "Activity Main Venue Page", such as the displayed text "Main Venue" and the control type "Button".

[0091] It should be noted that, for each operation description, the execution rule generation model can also set an exception handling strategy for the atomic operation. The exception handling strategy includes, for example, a preset threshold for retry, a timeout setting, and an exception rollback strategy. This allows the task execution end to consider the execution failure if the response time after executing the virtual operation reaches the set timeout time. In this case, the number of retries is less than or equal to the preset threshold. If the number of retries reaches the preset threshold, the exception rollback strategy will be used to transfer the request to a human agent.

[0092] Furthermore, the cloud can encapsulate the operation description sequence and the corresponding semantic features of each operation description, for example, according to the RPA protocol format, to obtain execution information. For example, the execution information can be data in JSON (JavaScript Object Notation) format, which can include the following fields: protocol version number, operation description sequence (operation type, semantic representation, timeout, and number of retries for each operation), and global configuration parameters (such as execution speed mode and exception handling strategy).

[0093] Because of the standardized format of the RPA protocol, the execution information encapsulated using the RPA protocol format can be adapted to different versions and operating systems of edge electronic devices, which helps to ensure that task execution terminals of different versions and system environments can parse and execute the planning information.

[0094] The above embodiments describe the embodiments of this application from the perspective of different execution ends. The following describes the automatic task execution method of the embodiments of this application from the perspective of the functional components of each end.

[0095] For example, taking the deployment of an IM client and an edge execution engine on the task execution end as an example, and the deployment of a cache gateway and an LLM inference model on the cloud as an example, the exemplary signaling interaction process of the above components during the implementation of the automatic task execution method is as follows: Figure 3 As shown.

[0096] Step S31: After receiving the natural language query input by the user, the IM client collects the basic information of the current session and transmits the query and basic information to the client-side execution engine.

[0097] The query could be something like "How to claim discount coupons for year-end promotions". Basic information could include user ID, app version number (e.g., v10.2.0), and promotional activity identifier (e.g., "Year-end mega sale").

[0098] In step S32, the execution engine on the client side generates a query key based on the basic information and searches for an RPA protocol packet that matches the query key in the local cache. If an RPA protocol packet is found, proceed to step S38; otherwise, proceed to step S33.

[0099] The query key could be something like "hash("Get Coupons")_v10.2.0_Year-End Sale". For example, an RPA protocol packet refers to a data packet encapsulating execution information.

[0100] Step S33: Send a query request to the cache gateway.

[0101] The query request includes both the query and basic information.

[0102] Step S34: The cache gateway searches for an RPA protocol packet that matches the query key from the CDN edge storage node. If an RPA protocol packet is found, it is sent to the end-side execution engine. If no RPA protocol packet is found, step S35 is executed.

[0103] Step S35: Transfer the Query and basic information to the LLM inference model.

[0104] Step S36: The LLM inference model generates an RPA protocol packet based on the query and basic information.

[0105] For example, the LLM inference model can combine the query and basic information to deduce the thought chain, identify the user's intent type as "claim coupons", and then retrieve the matching page navigation topology based on the current APP version number and activity identifier. Based on the intent type and page navigation topology, it infers and generates an atomic operation sequence: [Jump to homepage] -> [Click the "Year-end promotion main venue" entrance] -> [Locate the "Discount Coupon" area] -> [Click the "Claim Now" button]. After that, the atomic operation sequence and related semantic selectors are encapsulated into an RPA protocol packet.

[0106] For details, please refer to the relevant descriptions in the above embodiments, which will not be elaborated here.

[0107] In step S37, the LLM inference model distributes RPA protocol packets to the caching gateway and the end-side execution engine.

[0108] The caching gateway can store received RPA protocol packets to CDN edge storage nodes.

[0109] In step S38, the execution engine on the client side renders the execution confirmation card and displays it through the IM client.

[0110] The confirmation card will display information such as: "We have detected that you want to claim a discount coupon for the big promotion. I will automatically redirect you to the event page and ask you to click to claim it. Do you want to continue?" The card will also provide "Allow" and "Cancel" buttons below it.

[0111] In step S39, in response to receiving the user's permission to execute, the terminal execution engine automatically executes the coupon claiming task according to the RPA protocol packet.

[0112] For example, the client-side execution engine can capture the current page of the app and construct a view tree according to the operation description order in the RPA protocol. After determining the starting operation page view, for any view, a semantic selector is used to match the target interface element in the view. By performing the corresponding virtual operation on the target interface element, the app is automatically triggered to update the GUI, thereby realizing the automatic execution of the coupon redemption task.

[0113] For details, please refer to the relevant descriptions in the above embodiments, which will not be elaborated here.

[0114] Step S310: After the execution engine on the client side completes its execution, it sends the execution result to the IM client.

[0115] The execution result may include, for example, a screenshot of the current "Successfully Claimed" status page, and a result description text such as "You have successfully claimed the discount coupon, which can be viewed in your wallet."

[0116] Understandable. Figure 3 The illustration is merely illustrative and does not constitute a limitation on the embodiments of this application. The embodiments of this application may also include comparative... Figure 3 More or fewer steps. If other issues arise during execution, such as the failure of an atomic operation, retries can also be included. This will not be elaborated upon here.

[0117] In summary, upon receiving a task query, the task execution end obtains execution information containing a sequence of operation descriptions and automatically executes the corresponding operations on the client to be operated according to this execution information, thereby directly obtaining the task execution result. It is evident that in this process, the user does not need to manually follow the steps; the entire task operation process is completed automatically, effectively reducing the user abandonment rate due to cumbersome operation steps. Furthermore, the execution information is generated by the cloud based on basic information related to the task query, and this basic information is related to the client to be operated. In other words, the execution information matches the client to be operated, thus avoiding the problem of operation failure caused by mismatch between the execution information and the interface layout of the client to be operated. This is beneficial for improving the processing efficiency and user experience of operation guidance-type consultations. Moreover, in the technical solution of this application, the generation by the cloud-based model helps reduce the computational overhead on the task execution end, shortening the time from when the user initiates a query to when the execution information is obtained, thereby reducing the user's waiting time.

[0118] The above embodiments describe various implementations of the method provided in this application from the perspectives of collecting basic information, identifying intent types, encapsulating execution information, and automatically executing operation tasks based on the view tree. It should be understood that, corresponding to the above processing steps, the embodiments of this application can implement the above functions in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0119] For example, if the above implementation steps are achieved through software modules to realize the corresponding functions. Figure 4A As shown, an automatic task execution device is provided, which may include a determining module 411, an acquiring module 412, and an operating module 413. This automatic task execution device can be used to perform the aforementioned tasks. Figures 2A to 3 Some or all of the operations performed on the task execution end.

[0120] For example: the determination module 411 is used to determine basic information related to the task query information in response to receiving the task query information; the acquisition module 412 is used to acquire execution information matching the basic information, the execution information including an operation description sequence; the execution information is pre-generated by the cloud-based execution rule generation model based on the basic information; the operation module 413 is used to perform an operation on the client to be operated according to the operation description sequence to obtain the task execution result corresponding to the task query information.

[0121] Optionally, the acquisition module 412 is further configured to search for execution information matching the basic information from local storage; if the execution information is found, read the execution information; if the execution information is not found, send a search request carrying the basic information to the cloud, and receive the execution information from the cloud, wherein the execution information is searched or generated by the cloud.

[0122] Optionally, the acquisition module 412 is further configured to receive the execution information from the cache gateway, the execution information being retrieved by the cache gateway from the edge storage node in response to the lookup request; the acquisition module 412 is further configured to receive the execution information from the execution rule generation model, the execution information being generated by the execution rule generation model based on the basic information when the cache gateway does not find the execution information.

[0123] Optionally, the operation module 413 is further configured to construct a page view sequence corresponding to each operation description in the operation description sequence; determine a page view in the page view sequence that matches the current page of the client to be operated, and use it as the starting operation page view; and execute the corresponding operations sequentially according to the order of the page view sequence, starting from the starting operation page view.

[0124] Optionally, the operation module 413 is further configured to determine a page view in the page view sequence that is the same as the current page; if there is a page view in the page view sequence that is the same as the current page, determine the page view as the starting operation page view; if there is no page view in the page view sequence that is the same as the current page, determine the first page view in the page view sequence as the starting operation page view.

[0125] Optionally, the execution information also includes semantic representations of the interface elements corresponding to each operation description. The operation module 413 is further configured to, for any page view, determine the interface elements in the page view based on the semantic representation corresponding to the page view; and perform matching virtual operations on the interface elements according to the operation description corresponding to the page view.

[0126] Optionally, the automatic task execution device further includes a detection module and a transfer module. The detection module is used to detect the total number of repeated operations in response to the failure of the operation on the interface element. The operation module 413 is also used to repeatedly execute the virtual operation on the interface element and increment the total number by 1 if the total number is less than a preset threshold. The transfer module is used to transfer to a human operator if the total number is equal to the preset threshold.

[0127] Optionally, the automatic task execution device further includes a display module, which is used to display automatic execution authorization information, the authorization information including the operation description sequence; the operation module 413 is also used to perform operations on the client to be operated according to the operation description sequence if an instruction to grant permissions is received.

[0128] like Figure 4B As shown, an automatic task execution device is provided, which may include a calling module 421 and a sending module 422. This automatic task execution device can be used to perform the aforementioned tasks. Figures 2A to 3 Some or all of the operations in the cloud.

[0129] For example, module 421 is used to invoke the execution rule generation model in response to receiving basic information to generate execution information based on the basic information; the execution information includes an operation description sequence; the basic information is determined by the task execution end after receiving the task query information; and module 422 is used to send the execution information to the task execution end so that the task execution end performs an operation on the client to be operated according to the operation description sequence to obtain the task execution result corresponding to the task query information.

[0130] Optionally, the calling module 421 is further configured to identify the intent type and activity identifier corresponding to the task query information, as well as the version number of the proxy operation client; obtain the page navigation topology map corresponding to the activity identifier under the version number from a preset page navigation topology map library; generate the operation description sequence according to the intent type and the page navigation topology map; determine the semantic representation of the interface element corresponding to each operation description; and encapsulate the operation description sequence and the semantic representation of the interface element corresponding to each operation description according to a preset format to obtain the execution information.

[0131] Optionally, module 421 is further configured to, in response to receiving a lookup request, invoke the cache gateway to look up the execution information from the edge storage node based on the basic information carried in the lookup request, wherein the lookup request is sent by the task execution end when the execution information is not found in the local storage; if the execution information is not found in the edge storage node, invoke the execution rule generation model to generate execution information based on the basic information.

[0132] Optionally, module 421 is also used to call the cache gateway to store the execution information to the edge storage node.

[0133] The functions of each module in the devices of this application embodiment can be found in the corresponding descriptions of the methods described above, and they have corresponding beneficial effects, which will not be repeated here. It is understood that the device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components illustrated as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application solution according to actual needs.

[0134] Please refer to Figure 5 , Figure 5 An electronic device is provided, which can be implemented as carrying Figure 1 This is a task execution terminal or a cloud-based device. The electronic device may include a processor 51, a transceiver 52, and a memory 53, which are connected and communicate via a communication bus 54. The transceiver 52 can be used to communicate with the other end to obtain information and data. The memory 53 is used to store computer programs and data generated during the automatic execution of tasks. When the computer program is invoked, it causes the processor 51 to execute the aforementioned... Figures 2A to 3 The operation of some or all of the electronic devices.

[0135] For details on the implementation process, please refer to [link / reference]. Figures 2A to 3 The relevant descriptions are omitted here.

[0136] It should be understood that, in some feasible implementations, Figure 5 The processor illustrated can be a central processing unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. Figure 5The memory illustrated may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store information such as the device type.

[0137] This application also provides a computer-readable storage medium storing instructions for automatic task execution, which, when run on a computer, cause the computer to perform the aforementioned actions. Figures 2A to 3 Some or all of the steps in the method described in the illustrated embodiments.

[0138] This application also provides a computer program product that includes instructions for automatically executing tasks. When run on a computer, it causes the computer to perform the aforementioned actions. Figures 2A to 3 Some or all of the steps in the method described in the illustrated embodiments.

[0139] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0140] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0141] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0142] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0143] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, smartphone, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0144] Although alternative embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make further changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0145] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of this application should be included within the scope of protection of this invention.

Claims

1. A method for automatically executing tasks, characterized in that, The method includes: In response to receiving a task query message, determine the basic information related to the task query message; Obtain execution information matching the basic information, the execution information including an operation description sequence; the execution information is pre-generated by the cloud-based execution rule generation model based on the basic information; The operation is performed on the client to be operated according to the operation description sequence to obtain the task execution result corresponding to the task query information.

2. The method according to claim 1, characterized in that, The step of obtaining execution information that matches the basic information includes: Retrieve execution information from local storage that matches the basic information; If the execution information is found, read the execution information; If the execution information is not found, a search request carrying the basic information is sent to the cloud, and the execution information is received from the cloud, which is either searched or generated by the cloud.

3. The method according to claim 2, characterized in that, Receiving the execution information from the cloud includes: Receive the execution information from the cache gateway, the execution information being retrieved from the edge storage node by the cache gateway in response to the lookup request; or, The execution information is received from the execution rule generation model, which is generated by the execution rule generation model based on the basic information when the cache gateway does not find the execution information.

4. The method according to claim 1, characterized in that, The step of performing operations on the client to be operated according to the operation description sequence includes: Construct a sequence of page views corresponding to each operation description in the operation description sequence; The page view that matches the current page of the client to be operated in the page view sequence is determined as the starting operation page view; Starting from the initial operation page view, the corresponding operations are executed sequentially according to the order of the page view sequence.

5. The method according to claim 4, characterized in that, The step of determining the page view in the page view sequence that matches the current page of the client to be operated, and using it as the starting operation page view, includes: Identify the page view in the page view sequence that is identical to the current page; If there is a page view in the page view sequence that is the same as the current page, that page view will be determined as the starting operation page view; If there is no page view in the page view sequence that is the same as the current page, the first page view in the page view sequence is determined as the starting operation page view.

6. The method according to claim 4, characterized in that, The execution information also includes semantic representations of the interface elements corresponding to each operation description. The step of executing the corresponding operations sequentially according to the order of the page view sequence, starting from the initial operation page view, includes: For any given page view, determine the interface elements in the page view based on the semantic representation corresponding to the page view; Perform matching virtual operations on the interface elements according to the operation description corresponding to the page view.

7. The method according to claim 6, characterized in that, Also includes: In response to a failed operation on the interface element, the total number of repeated operations is detected; If the total number of times is less than a preset threshold, the virtual operation is repeated on the interface element, and the total number of times is incremented by 1. If the total number of attempts equals the preset threshold, the call is transferred to a human agent.

8. The method according to any one of claims 4-7, characterized in that, Also includes: Display automatic execution authorization information, the authorization information including the operation description sequence; If an instruction to grant permissions is received, the operation is performed on the client to be operated according to the operation description sequence.

9. A method for automatically executing a task, characterized in that, The method includes: In response to receiving basic information, the execution rule generation model is invoked to generate execution information based on the basic information; the execution information includes an operation description sequence; the basic information is determined by the task execution end after receiving the task query information; The execution information is sent to the task execution terminal so that the task execution terminal performs operations on the client to be operated according to the operation description sequence, so as to obtain the task execution result corresponding to the task query information.

10. The method according to claim 9, characterized in that, The call execution rule generation model generates execution information based on the basic information, including: Identify the intent type and activity identifier corresponding to the task query information, as well as the version number of the proxy operation client; Retrieve the page navigation topology map corresponding to the activity identifier under the version number from the preset page navigation topology map library; The operation description sequence is generated based on the intent type and the page navigation topology. Determine the semantic representation of the interface elements corresponding to each operation description; The execution information is obtained by encapsulating the operation description sequence and the semantic representation of the interface elements corresponding to each operation description in a preset format.

11. The method according to claim 9, characterized in that, Before invoking the execution rule generation model to generate execution information based on the basic information, the process also includes: In response to receiving a lookup request, the cache gateway is invoked to search for the execution information from the edge storage node based on the basic information carried in the lookup request. The lookup request is sent by the task execution end when the execution information is not found in the local storage. If the execution information is not found in the edge storage node, the execution rule generation model is invoked to generate execution information based on the basic information.

12. The method according to any one of claims 9-11, characterized in that, Also includes: The cache gateway is invoked to store the execution information to the edge storage node.

13. A task automatic execution system, characterized in that, The system includes a task execution terminal and a cloud platform, wherein the cloud platform deploys an execution rule generation model. The task execution terminal is used to determine basic information related to the task query information after receiving the task query information; The cloud platform is used to invoke the execution rule generation model to generate execution information based on the basic information, and the execution information includes an operation description sequence. The task execution terminal is also used to obtain the execution information that matches the basic information, and to perform operations on the client to be operated according to the operation description sequence, so as to obtain the task execution result corresponding to the task query information.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the program to implement the method as described in any one of claims 1-12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by a processor to implement the method as described in any one of claims 1-12.

16. A computer program product, comprising a computer program, characterized in that, The computer program is executed by a processor to implement the method according to any one of claims 1-12.

Images