Method and device for executing user operation, electronic equipment and storage medium
By constructing hybrid operation sequences and combining the use of rule engines and large language model engines, the problems of low flexibility and efficiency in dynamic processing in browser automation technology are solved, achieving efficient and flexible browser operation processing.
Patent Information
- Application Number
- CN202511724323.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-23
- Publication Date
- 2026-02-17
AI Technical Summary
Existing browser automation technologies struggle to balance the efficiency of rule scripts with the dynamic processing flexibility of large language models. Rule scripts are less flexible in handling dynamic changes, while LLMs are less efficient in scenarios with high real-time requirements.
Construct a hybrid operation sequence, including deterministic and non-deterministic operations. Utilize the browser's rule engine to execute deterministic operations and the large language model engine to execute non-deterministic operations. Combine scheduled tasks and exception handling mechanisms to improve dynamic processing capabilities.
It achieves the dynamic processing flexibility of a large language model while maintaining the efficiency of rule scripts, reduces the dependence on LLM, and improves operational efficiency and real-time performance.
Smart Images
Figure CN121542528A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and more specifically to a method, apparatus, electronic device, and storage medium for performing user operations. Background Technology
[0002] With the development of computer technology, browsers have become an important tool for users to access the Internet, and browser automation technology has been widely used in various fields. Current browser automation technology mainly relies on rule scripts or Large Language Models (LLMs). However, with the continuous development of web technology, fixed rule scripts cannot handle dynamically changing operational logic and have poor processing flexibility; while LLMs require a long time for inference, making them difficult to meet the needs of real-time operation scenarios and resulting in poor processing efficiency.
[0003] Therefore, there is an urgent need for a method that balances the efficiency of rule scripts with the dynamic processing flexibility of large language models. Summary of the Invention
[0004] In view of this, embodiments of the present invention provide a method, apparatus, electronic device, and storage medium for performing user operations, so as to balance the efficiency of rule scripts and the dynamic processing flexibility of large language models.
[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:
[0006] The first aspect of this invention discloses a method for performing user operations, the method comprising:
[0007] Construct a hybrid operation sequence, which includes deterministic operations and uncertain operations;
[0008] Obtain the target operation from the mixed operation sequence;
[0009] If the target operation is a deterministic operation, the target operation is executed through the browser's rule engine;
[0010] If the target operation is an uncertain operation, the target operation is executed through the browser's large language model engine.
[0011] Preferably, constructing a hybrid operation sequence includes:
[0012] Capture all user actions in the browser to obtain deterministic actions;
[0013] Obtain the natural language description input by the user in the interactive page of the browser that needs to be executed by the large language model, so as to obtain the uncertain operation;
[0014] A hybrid operation sequence is constructed using the obtained deterministic and uncertain operations.
[0015] Preferably, if the target operation is the uncertain operation, executing the target operation through the browser's large language model engine includes:
[0016] Obtain page information of the page in the browser that is related to the target operation;
[0017] The page information and the target operation are input into the browser's large language model engine to obtain a fixed-format operation returned by the large language model engine;
[0018] The browser's rules engine executes operations in a fixed format as fed back by the large language model engine.
[0019] Preferred options also include:
[0020] When the target operation fails, the context information generated by the failure is captured.
[0021] The context information is sent to the large language model engine to obtain the correction scheme for the target operation fed back by the large language model;
[0022] If the proposed correction passes the validity verification, the proposed correction is executed, and the process returns to the step of executing the target operation through the browser's rule engine, or the process returns to the step of executing the target operation through the browser's large language model engine.
[0023] If the proposed correction fails the validity verification, an error message will be output.
[0024] Preferably, before obtaining the target operation from the mixed operation sequence, the method further includes:
[0025] Create a scheduled task;
[0026] When the scheduled task is triggered, the step of obtaining the target operation from the mixed operation sequence is executed.
[0027] A second aspect of this invention discloses an apparatus for performing user operations, the apparatus comprising:
[0028] A construction unit is used to construct a hybrid operation sequence, which includes deterministic operations and uncertain operations;
[0029] An acquisition unit is used to acquire a target operation from the mixed operation sequence;
[0030] The first execution unit is configured to execute the target operation through the browser's rule engine if the target operation is the deterministic operation.
[0031] The second execution unit is used to execute the target operation through the browser's large language model engine if the target operation is the uncertain operation.
[0032] Preferably, the building unit includes:
[0033] The capture module is used to capture all user actions in the browser to obtain deterministic behavior;
[0034] The acquisition module is used to acquire the natural language description that the user inputs in the interactive page of the browser, which needs to be executed by the large language model, in order to obtain uncertain operations;
[0035] A construction module is used to construct a hybrid operation sequence using the obtained deterministic and uncertain operations.
[0036] Preferably, the second execution unit is specifically used to: obtain page information of the page in the browser related to the target operation;
[0037] The page information and the target operation are input into the browser's large language model engine to obtain a fixed-format operation returned by the large language model engine;
[0038] The browser's rules engine executes operations in a fixed format as fed back by the large language model engine.
[0039] A third aspect of the present invention discloses a computer device, comprising: a processor and a memory, the processor and the memory being connected via a bus; wherein, the processor is used to call and execute a program stored in the memory; the memory is used to store the program, the program being used to implement the method for performing user operations disclosed in the first aspect of the present invention.
[0040] A fourth aspect of the present invention discloses a storage medium storing computer-executable instructions for executing the method for performing user operations disclosed in the first aspect of the present invention.
[0041] Based on the above embodiments of the present invention, a method, apparatus, electronic device, and storage medium for executing user operations are provided. The method comprises: constructing a hybrid operation sequence, which includes deterministic operations and uncertain operations; obtaining a target operation from the hybrid operation sequence; if the target operation is a deterministic operation, executing the target operation through the browser's rule engine; if the target operation is an uncertain operation, executing the target operation through the browser's large language model engine. This solution constructs a hybrid operation sequence including deterministic and uncertain operations, uses a rule engine to execute the deterministic operations in the hybrid operation sequence, and uses a large language model engine to execute the uncertain operations in the hybrid operation sequence, thus maintaining the efficiency of rule scripts while possessing the dynamic processing flexibility of large language models. Attached Figure Description
[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0043] Figure 1 A flowchart illustrating a method for performing user operations, provided as an embodiment of the present invention;
[0044] Figure 2 This is an example diagram illustrating the implementation architecture of a method for performing user operations, provided in an embodiment of the present invention.
[0045] Figure 3 A flowchart illustrating a method for performing user operations according to an embodiment of the present invention;
[0046] Figure 4 A flowchart illustrating the exception handling mechanism provided in this embodiment of the invention;
[0047] Figure 5 This is a structural block diagram of a device for performing user operations, provided as an embodiment of the present invention. Detailed Implementation
[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0049] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0050] To better understand the following content, some terms will be explained first.
[0051] XPath: A path expression for precisely locating elements in an HTML document.
[0052] DOM (Document Object Model): A structured representation of front-end page elements, representing an HTML or XML document as a tree structure, where each node is an element in the document, such as a tag, attribute, or text.
[0053] Reasoning-Action Framework: An AI framework that combines reasoning and action to enhance the processing capabilities of complex tasks through dynamic interaction.
[0054] With the development of computer technology, browsers have become an important tool for users to access the Internet. Browser automation technology is widely used in fields such as web page testing, data scraping, and automated operations. The core goal of browser automation technology is to simulate user operations in the browser in a programmatic way, thereby improving efficiency and reducing human intervention.
[0055] Research has found that current browser automation technologies mainly rely on rule scripts or large language models (LLMs).
[0056] Browser automation techniques relying on rule-based scripts: These techniques use predefined scripts (such as XPath and CSS selectors) from browser automation tools to locate web page elements and perform operations. The advantages of this approach are speed and low cost, making it suitable for handling web pages with fixed logic. However, with the increasing prevalence of dynamic web pages (such as single-page applications and dynamically loaded content), the limitations of rule-based scripts are becoming increasingly apparent. On the one hand, rule-based scripts cannot handle dynamically changing logic, such as scenarios where operations cannot be dynamically adjusted based on real-time data, resulting in poor flexibility. On the other hand, when the web page structure changes, the rule-based scripts need to be manually modified, leading to high maintenance costs.
[0057] Browser automation technology relying on LLM (Library Management Model) executes operations by inputting tasks described in natural language into the LLM. However, on the one hand, calling the LLM consumes a lot of computing resources, significantly increasing the operating cost; on the other hand, the inference process of the LLM takes a long time, making it difficult to meet the needs of scenarios with high real-time requirements.
[0058] In summary, current browser automation technologies struggle to balance flexibility and efficiency. Therefore, there is an urgent need for a method that can preserve the efficiency and low cost of rule scripts while also being able to handle dynamic logic.
[0059] To address this, this invention proposes a method, apparatus, electronic device, and storage medium for executing user operations. It constructs a hybrid operation sequence including deterministic and uncertain operations, uses a rule engine to execute the deterministic operations in the hybrid operation sequence, and uses a large language model engine to execute the uncertain operations in the hybrid operation sequence. This approach maintains the efficiency of rule scripts while possessing the dynamic processing flexibility of large language models.
[0060] See Figure 1 The diagram illustrates a flowchart of a method for performing user operations according to an embodiment of the present invention, the method comprising:
[0061] Step S101: Construct a mixed operation sequence.
[0062] In the specific implementation step S101, all user operations in the browser are captured to obtain deterministic operations. Natural language descriptions input by the user on the browser's interactive page, which require execution by the large language model, are obtained to obtain indeterminate operations. A hybrid operation sequence is constructed using the obtained deterministic and indeterminate operations; that is, the constructed hybrid operation sequence includes both deterministic and indeterminate operations.
[0063] It should be noted that the information for deterministic operations includes at least: the type of operation (such as click operation and input operation), the page URL, the XPath of the element, and the attributes of the element.
[0064] Specifically, users open the operation interface through a browser plugin and click the record button to activate the browser's operation recording function. After activating the operation recording function, the browser plugin automatically captures all the user's operations in the browser. The user's operations in the browser are deterministic operations (such as clicking a fixed XPath element). The deterministic operations are added to the operation sequence according to the order of each operation.
[0065] When the user clicks the stop recording button, an "LLM execution node" is added to the browser plugin's interactive page. Then, the user enters a natural language description of what LLM needs to execute in the pop-up input box. The "natural language description of what LLM needs to execute" entered by the user is the uncertain operation. The uncertain operation is added to the operation sequence to construct a hybrid operation sequence.
[0066] It is understandable that the deterministic operations in the hybrid operation sequence are equivalent to "rule nodes," while the non-deterministic operations are equivalent to "LLM execution nodes." The constructed hybrid operation sequence is then stored.
[0067] Step S102: Obtain the target operation from the mixed operation sequence.
[0068] In the specific implementation step S102, the target operation is obtained from the mixed operation sequence. The target operation is either a deterministic operation or an uncertain operation that needs to be performed.
[0069] If the target operation is a deterministic operation, proceed to step S103; if the target operation is an uncertain operation, proceed to step S104.
[0070] Step S103: If the target operation is a deterministic operation, execute the target operation through the browser's rule engine.
[0071] In the specific implementation of step S103, if the target operation is a deterministic operation, the browser's rule engine is used to execute the target operation directly.
[0072] Step S104: If the target operation is an uncertain operation, execute the target operation through the browser's large language model engine.
[0073] In the specific implementation of step S104, if the target operation is an uncertain operation, obtain the page information of the page related to the target operation in the browser. The page information is the DOM object tree of the page elements. The page information includes at least the content, type and XPath of the element.
[0074] Specifically, if the target operation is uncertain, the browser plugin's built-in code is used to obtain page information related to the target operation.
[0075] The page information and the target operation (which is the natural language description entered by the user) are input into the browser's Large Language Model (LLM) engine to obtain a fixed-format operation from the LLM engine. This fixed-format operation is equivalent to a structured operation instruction (which can indicate subsequent operations). For example, the fixed-format operation is: Click the lowest-priced item → Click the lowest-priced item → Calculate the minimum value → Click the corresponding element. The browser's rule engine then executes the fixed-format operation returned by the LLM engine.
[0076] In some embodiments, to improve work efficiency and achieve automated testing, this solution provides a scheduled task mechanism. Specifically, a scheduled task is created, and when the scheduled task is triggered, step S102 is executed.
[0077] The specific way to create a scheduled task is to use "chrome.alarms.create()" in the extended background script (background.js) to name the scheduled task and configure its parameters.
[0078] The naming of the scheduled task is used to set a unique name for the scheduled task. The parameter configuration of the scheduled task mainly includes the following: the delay time for the first trigger (minutes), the interval time for repeated triggers (minutes), and specifying the precise trigger timestamp (milliseconds).
[0079] Register a listener in the background script. Specifically, use "chrome.alarms.onAlarm.addListener()" to capture the event that triggers the scheduled task. Get the name of the triggered scheduled task through the callback function's parameter, and perform related operations (such as sending network requests, updating local data, and sending notifications) in the listener. In this way, the browser can use the real-time interface to implement cross-session, low-power scheduled tasks.
[0080] In some embodiments, the target operation may fail during execution, so this solution provides an exception handling mechanism.
[0081] When a target operation fails, the context information generated by the failure is captured. Specifically, when a target operation fails, the error information generated by the failure (such as missing elements, page crashes, etc.) is captured, and then the context information is constructed using this error information.
[0082] Contextual information is sent to the large language model engine to obtain feedback from the large language model on the correction scheme for the target operation.
[0083] If the revised solution passes the validity verification, execute the revised solution and return to step S103 or step S104. Specifically, if the revised solution passes the validity verification and the target operation that failed to execute is a deterministic operation, then execute the revised solution and return to step S103; if the revised solution passes the validity verification and the target operation that failed to execute is an uncertain operation, then execute the revised solution and return to step S104.
[0084] If the revised solution fails the validity verification, an error message will be output.
[0085] In this embodiment of the invention, a hybrid operation sequence including deterministic and uncertain operations is constructed. The deterministic operations in the hybrid operation sequence are executed using a rule engine, and the uncertain operations in the hybrid operation sequence are executed using a large language model engine. This maintains the efficiency of the rule script while possessing the dynamic processing flexibility of the large language model.
[0086] In practical applications, such as Figure 2 As shown in the example architecture diagram, this solution is implemented by the following four-layer architecture: user interface layer 201, core processing layer 202, large model interaction layer 203, and data layer 204.
[0087] User Interface Layer 201: Provides a visual interface to users through a browser plugin. The visual interface mainly includes interfaces for scheduled task management, mixed operation sequence editing, and scheduled task settings.
[0088] The core processing layer 202 has the following functions: recording user operations, generating mixed operation sequences, execution engines (rule engine and LLM engine), and exception handling mechanisms.
[0089] The large model interaction layer 203 mainly includes the following functions: web page element capture, LLM inference, and execution of LLM inference.
[0090] Data layer 204 has the following functions: storing mixed operation sequences, storing LLM configurations, and storing scheduled tasks.
[0091] In the above Figure 2 Based on the illustrated architecture diagram, one application scenario of this solution is as follows: Figure 3 A flowchart illustrating a method for performing user operations is shown below. Figure 3 include:
[0092] Step S301: Record user actions via browser plugin.
[0093] Step S302: Construct a mixed operation sequence.
[0094] In the specific implementation step S302, the recorded user operations are added to the mixed operation sequence as deterministic operations, and the natural language description entered by the user during the process of adding the "LLM execution node" is added to the mixed operation sequence as an uncertain operation.
[0095] Step S303: Execute the mixed operation sequence.
[0096] In the specific implementation of step S303, the operations in the mixed operation sequence are executed using a dynamic routing method. Specifically, for deterministic operations in the mixed operation sequence, the rule engine is used to directly execute the deterministic operations; for uncertain operations in the mixed operation sequence, the large language model engine is used to execute the uncertain operations.
[0097] It is understandable that the browser plugins mentioned in the above embodiments can be mainly divided into the following parts: "operation page", "operation listening and recording mechanism", "adding uncertain operations", "database and scheduled task management", and "scheduled task configuration".
[0098] Operation page section: The interaction logic is displayed on the browser plugin's homepage. Specifically, users can add, modify, or delete scheduled tasks through the task management bar on the homepage, construct mixed operation sequences (recording user deterministic operations and obtaining user-added uncertain operations) through the homepage, and select the LLM used by the large language model engine, configure LLM parameters, fill in API_KEY, and fill in the URL whitelist through the homepage.
[0099] It should be noted that when configuring LLM parameters and filling in the API_KEY, the LLM prompt words can be set according to the actual situation. These prompt words can specify response rules (e.g., requiring responses in JSON format).
[0100] Operation monitoring and recording mechanism: When a user creates a new scheduled task and clicks the record button, the operation is captured. Specifically, the user's operations (deterministic operations) are captured in real time through the DOM monitoring function, and the deterministic operations are added to the mixed operation sequence in order.
[0101] Adding an uncertain operation section: The user clicks the stop recording button, then clicks to add an "LLM execution node" and adds a natural language description (uncertain operation) to the "LLM execution node," thus adding the uncertain operation to the mixed operation sequence.
[0102] Database and Scheduled Task Management: Uses an indexed database to store mixed operation sequences. The storage records for mixed operation sequences include: sequence name, creation time, and operation.
[0103] Scheduled task configuration section: Based on the browser's scheduled interface, scheduled triggers are implemented, supporting the configuration of recurring scheduled tasks (such as daily or weekly recurrence). The purpose of scheduled tasks is to automatically perform some repetitive tasks (regularly check emails, update data, download files), saving users time and effort and improving work efficiency. Scheduled tasks can also be used to automatically test the website's functions, ensuring the stability and performance of the website at different times.
[0104] It should be noted that the browser's timer interface allows scheduled tasks to run in the browser background. Even if the page is closed or the browser is restarted, the scheduled task can still be triggered, and the information of the scheduled task is stored inside the browser and can be automatically restored after the browser is restarted (controlled by the persistent parameter).
[0105] To better understand the exception handling mechanism proposed in this solution, through Figure 4 The flowchart illustrating the exception handling mechanism is shown below. Figure 4 Includes the following steps:
[0106] Step S401: Perform the target operation.
[0107] Step S402: Determine whether the target operation was executed successfully. If successful, proceed to step S403. If unsuccessful, proceed to step S404.
[0108] Step S403: Set the next operation as the target operation and return to step S401.
[0109] Step S404: Capture error information.
[0110] In the specific implementation of step S404, if the target operation fails, the browser is manipulated through code injection to capture underlying errors (such as missing elements and page crashes) and obtain error information.
[0111] Step S405: Construct context information.
[0112] Step S406: Send the context information to the large language model engine.
[0113] Step S407: Obtain the correction plan.
[0114] In the specific implementation step S407, a correction plan is obtained for the failure of the target operation.
[0115] For example, the corrective action could be: if the operation times out or the element is lost, trigger a retry or replacement operation.
[0116] Step S408: Verify the correction plan. If the correction plan is valid, return to step S401; if the correction plan is invalid, terminate and output an exception message.
[0117] To better understand the process from steps S401 to S408, let's take the scenario of "login failure" as an example: After executing "enter password", the page redirects to the verification code; the verification code element is detected to be missing and there is no password box; the LLM determines that "login was interrupted by verification code" and triggers the exception handling mechanism; the generated correction solution is: try to call the verification code resolution tool, or mark "UNRECOVERABLE_FAILURE", save a snapshot and terminate; repeat the above steps.
[0118] In summary, this solution categorizes operations into deterministic operations and uncertain operations described in natural language, enabling flexible handling of dynamic logic. The rule engine directly executes deterministic operations, while the LLM engine transforms uncertain operations into fixed-format operations in real time, thus adapting to dynamically changing scenarios. The LLM is only invoked when processing uncertain operations, reducing reliance on it and lowering computational resources and costs. The rule engine processes deterministic operations faster than the LLM engine, while the LLM engine only handles uncertain operations, reducing execution time and improving efficiency.
[0119] Corresponding to the method for performing user operations provided in the above embodiments of the present invention, see also... Figure 5 The present invention also provides a structural block diagram of a device for performing user operations, the device including: a construction unit 501, an acquisition unit 502, a first execution unit 503, and a second execution unit 504.
[0120] Construction unit 501 is used to construct a mixed operation sequence, which includes deterministic operations and uncertain operations.
[0121] The acquisition unit 502 is used to acquire the target operation from the mixed operation sequence.
[0122] The first execution unit 503 is used to execute the target operation through the browser's rule engine if the target operation is a deterministic operation.
[0123] The second execution unit 504 is used to execute the target operation through the browser's large language model engine if the target operation is an uncertain operation.
[0124] In a specific implementation, the second execution unit 504 is specifically used to: obtain page information of the page related to the target operation in the browser; input the page information and the target operation into the browser's large language model engine to obtain a fixed-format operation fed back by the large language model engine; and execute the fixed-format operation fed back by the large language model engine through the browser's rule engine.
[0125] In this embodiment of the invention, a hybrid operation sequence including deterministic and uncertain operations is constructed. The deterministic operations in the hybrid operation sequence are executed using a rule engine, and the uncertain operations in the hybrid operation sequence are executed using a large language model engine. This maintains the efficiency of the rule script while possessing the dynamic processing flexibility of the large language model.
[0126] Preferred, combined Figure 5 The content shown indicates that the construction unit 501 includes a capture module, an acquisition module, and a construction module. The execution principle of each module is as follows:
[0127] The capture module is used to capture all user actions in the browser to obtain deterministic behavior.
[0128] The acquisition module is used to acquire the natural language description that the user inputs in the browser's interactive page, which needs to be executed by the large language model, in order to obtain uncertain operations.
[0129] The building module is used to construct a mixed sequence of operations using the obtained deterministic and uncertain operations.
[0130] Preferred, combined Figure 5 The device, as shown, also includes:
[0131] An exception handling unit is used to capture the context information generated by the failure of the target operation when the failure of the target operation is detected; send the context information to the large language model engine to obtain the correction scheme for the target operation fed back by the large language model; if the correction scheme passes the validity verification, execute the correction scheme and return to the execution of the first execution unit 503 or the second execution unit 504; if the correction scheme fails the validity verification, output exception information.
[0132] Preferred, combined Figure 5 The device, as shown, also includes:
[0133] The task unit is used to create scheduled tasks. When a scheduled task is triggered, the acquisition unit 502 is executed.
[0134] Preferably, the present invention also provides a computer device, including: a processor and a memory, the processor and the memory being connected via a bus; wherein, the processor is used to call and execute a program stored in the memory; the memory is used to store the program, the program being used to implement the method for performing user operations provided in the above method embodiments.
[0135] Preferably, the present invention also provides a storage medium storing computer-executable instructions for performing the method for performing user operations provided in the above-described method embodiments.
[0136] In summary, the embodiments of the present invention provide a method, apparatus, electronic device, and storage medium for performing user operations, constructing a hybrid operation sequence including deterministic and uncertain operations, using a rule engine to execute the deterministic operations in the hybrid operation sequence, and using a large language model engine to execute the uncertain operations in the hybrid operation sequence, which can maintain the efficiency of rule scripts and have the dynamic processing flexibility of large language models.
[0137] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0138] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.
[0139] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method of performing a user operation, characterized by, The method comprises: constructing a mixed operation sequence comprising deterministic operations and uncertain operations; obtaining a target operation from the mixed operation sequence; if the target operation is the deterministic operation, executing the target operation through a rule engine of the browser; if the target operation is the uncertain operation, executing the target operation through a large language model engine of the browser.
2. The method of claim 1, wherein, constructing a mixed operation sequence, comprising: capturing all operations of a user in a browser to obtain deterministic operations; obtaining natural language descriptions input by the user in an interactive page of the browser and requiring execution by a large language model to obtain uncertain operations; constructing a mixed operation sequence using the obtained deterministic operations and uncertain operations.
3. The method of claim 1, wherein, if the target operation is the uncertain operation, executing the target operation through the large language model engine of the browser, comprising: obtaining page information of a page in the browser related to the target operation; inputting the page information and the target operation into the large language model engine of the browser to obtain operations in a fixed format fed back by the large language model engine; executing the operations in the fixed format fed back by the large language model engine through a rule engine of the browser.
4. The method of claim 1, wherein, further comprising: when it is detected that the target operation fails to be executed, capturing context information generated by the failure of the target operation to be executed; sending the context information to the large language model engine to obtain a correction scheme for the target operation fed back by the large language model; if the correction scheme passes an effectiveness verification, executing the correction scheme, returning to the step of executing the target operation through the rule engine of the browser, or returning to the step of executing the target operation through the large language model engine of the browser; if the correction scheme fails to pass the effectiveness verification, outputting exception information.
5. The method of claim 1, wherein, before the step of obtaining a target operation from the mixed operation sequence, further comprising: creating a timing task; when the timing task is triggered, executing the step of obtaining a target operation from the mixed operation sequence.
6. An apparatus for performing a user operation, the apparatus comprising: The device comprises: a construction unit configured to construct a mixed operation sequence comprising deterministic operations and uncertain operations; an obtaining unit configured to obtain a target operation from the mixed operation sequence; a first execution unit configured to execute the target operation through a rule engine of a browser if the target operation is the deterministic operation; a second execution unit configured to execute the target operation through a large language model engine of the browser if the target operation is the uncertain operation.
7. The apparatus of claim 6, wherein, The construction unit comprises: a capturing module configured to capture all operations of a user in a browser to obtain deterministic operations; an obtaining module configured to obtain natural language descriptions input by the user in an interactive page of the browser and requiring execution by a large language model to obtain uncertain operations; a construction module configured to construct a mixed operation sequence using the obtained deterministic operations and uncertain operations.
8. The apparatus of claim 6, wherein, The second execution unit is specifically configured to obtain page information of a page in the browser related to the target operation. input the page information and the target operation into a large language model engine of the browser to obtain an operation in a fixed format fed back by the large language model engine; execute the operation in the fixed format fed back by the large language model engine through a rule engine of the browser.
9. A computer device, comprising: Comprise: a processor and a memory connected through a bus; wherein the processor is configured to invoke and execute a program stored in the memory; the memory is configured to store a program, and the program is configured to implement the method for executing a user operation according to any one of claims 1-5.
10. A storage medium, characterized by The storage medium stores computer executable instructions, and the computer executable instructions are used to execute the method for executing a user operation according to any one of claims 1-5.