A natural language driven web page operation method and system

Abstract

The application relates to a natural language driven webpage operation method and system, and belongs to the technical field of webpage automation and human-computer interaction. The method comprises the following steps: receiving a natural language operation instruction input by a user; in response to the natural language operation instruction, calling a semantic understanding model to analyze structured document data of a target webpage, generating and executing an operation instruction sequence; in the process of executing the operation instruction sequence, recording operation information corresponding to each operation step in the operation instruction sequence; in response to a solidification instruction triggered by the user, encapsulating the recorded operation information into a solidified operation instruction set, and storing the solidified operation instruction set in association with the website information of the target webpage; and in response to a calling instruction of the stored solidified operation instruction set, matching the corresponding solidified operation instruction set according to the calling instruction, and executing the operation instruction sequence recorded in the solidified operation instruction set. The application has the effects of improving webpage operation efficiency and reducing resource consumption.

Description

Technical Field

[0001] This application relates to the fields of web page automation and interpersonal interaction technology, and in particular to a natural language-driven web page operation method and system. Background Technology

[0002] Currently, natural language-driven webpage operation methods are primarily implemented through artificial intelligence models. After a user inputs a natural language command, the system invokes the AI ​​model to parse the structured document data of the target webpage in real time, generating and executing a sequence of corresponding operation commands. This approach allows non-technical users to complete webpage operations through natural language interaction, without requiring programming skills or manually writing operation scripts.

[0003] However, in the aforementioned existing technologies, the system needs to call an artificial intelligence model to parse the structured document data of the webpage in real time each time an operation is performed, and only after generating operation instructions can they be executed. Specifically, even if the user needs to repeatedly perform the same operation on the same webpage, each operation consumes the computing resources and call costs of the artificial intelligence model. For high-frequency repetitive operation scenarios, resource consumption accumulates linearly, resulting in serious resource waste; therefore, improvements are needed. Summary of the Invention

[0004] To improve webpage operation efficiency and reduce resource consumption, this application provides a natural language-driven webpage operation method and system.

[0005] Firstly, this application provides a natural language-driven webpage operation method, including: Receive natural language operation instructions input by the user; In response to the natural language operation instructions, a semantic understanding model is invoked to parse the structured document data of the target webpage, generate a sequence of operation instructions to operate on the target webpage, and execute the sequence of operation instructions; During the execution of the operation instruction sequence, the operation information corresponding to each operation step in the operation instruction sequence is recorded; In response to a user-triggered fixed command, the recorded operation information is encapsulated into a fixed operation command set, and the fixed operation command set is associated with and stored with the URL information of the target webpage; In response to a user's call command for a stored set of solidified operation instructions, the corresponding set of solidified operation instructions is matched according to the call command, and the sequence of operation instructions recorded in the set of solidified operation instructions is executed.

[0006] By adopting the above technical solution, users do not need programming skills and can drive web page operations using only natural language. On the first execution, the semantic understanding model automatically parses and generates a sequence of operation instructions. During the execution process, the operation information is recorded and encapsulated into a fixed set of operation instructions. When the fixed set of operation instructions is called on the same web page later, the stored sequence of operation instructions is executed directly without calling the semantic understanding model for parsing again. This reduces the number of times the semantic understanding model is called, reduces the execution time of a single operation, and improves the execution speed of repeated operations to the millisecond level.

[0007] Optionally, the method further includes: When executing the sequence of operation instructions recorded in the solidified operation instruction set, if the operation information recorded in the solidified operation instruction set cannot locate the corresponding operation target in the structured document data of the current webpage, then the original natural language operation intent corresponding to the solidified operation instruction set is extracted. Based on the original natural language operation intent, the semantic understanding model is invoked to re-parse the structured document data of the current webpage, generate a new sequence of operation instructions, and replace the operation information recorded in the solidified operation instruction set with the new sequence of operation instructions.

[0008] By adopting the above technical solution, when changes occur in the structured document data of the target webpage, causing the operation information in the fixed operation instruction set to be unable to locate the corresponding operation target, the system extracts the original natural language operation intent corresponding to the fixed operation instruction set, calls the semantic understanding model to re-parse the changed webpage, generates a new operation instruction sequence, and replaces the original operation information. This solution allows the fixed operation instruction set to continue to be used after the webpage structure changes, without requiring the user to re-execute the initial operation process, thereby reducing the maintenance cost of the fixed operation instruction set and extending its effective use period.

[0009] Optionally, the method further includes: During the execution of the operation instruction sequence, dynamic data in the operation instruction sequence is identified, and the source path of the dynamic data is obtained, wherein the source path is used to indicate the way to obtain the current value of the dynamic data when the fixed operation instruction set is executed; In response to a user-triggered fixed instruction, the source path and the operation steps corresponding to the dynamic data are associated and stored in the fixed operation instruction set; When executing the solidified operation instruction set, the current value of the dynamic data is obtained in real time according to the source path, and the corresponding operation steps are executed.

[0010] By adopting the above technical solution, dynamic data is identified and its source path is obtained during the execution of the operation instruction sequence. During the solidification process, the source path, rather than the specific value, is stored in the solidified operation instruction set. In subsequent executions, the system obtains the current value of the dynamic data in real time based on the source path, instead of using the fixed value from the solidification process. This solution enables the solidified operation instruction set to adapt to data changes (such as prices, exchange rates, dates, etc., which change over time), ensuring that the latest and most valid data is entered each time, avoiding operational errors caused by using outdated data.

[0011] Optionally, the source path for obtaining the dynamic data includes: When the dynamic data corresponds to multiple source paths, the original natural language operation intent corresponding to the fixed operation instruction set is obtained, and the semantic matching degree between the original natural language operation intent and the data semantic tag associated with each source path is calculated. The source path with the highest semantic matching degree is taken as the source path of the dynamic data.

[0012] By adopting the above technical solution, when the same dynamic data corresponds to multiple candidate source paths, the system obtains the original natural language operation intent corresponding to the fixed operation instruction set, calculates the semantic matching degree between the intent and the data semantic tags associated with each source path, and selects the source path with the highest matching degree as the source path of the dynamic data. This solution can automatically select the data source that best matches the semantics of the current operation intent from multiple candidate data sources without requiring manual specification or intervention from the user, thereby improving the accuracy of dynamic data acquisition.

[0013] Optionally, the fixed operation instruction set also stores reference execution context information when the operation instruction sequence is executed; The execution of the sequence of operation instructions recorded in the solidification operation instruction set includes: Obtain the current execution context information and the reference execution context information corresponding to the fixed operation instruction set; verify whether the current execution context information matches the reference execution context information; If a match is found, the sequence of operation instructions recorded in the solidified operation instruction set is executed; If there is no match, the original natural language operation intent corresponding to the solidified operation instruction set is obtained. Based on the original natural language operation intent and the current execution context information, the structured document data of the current webpage is re-parsed to generate a new operation instruction sequence adapted to the current execution context information. The corresponding operation instruction sequence in the solidified operation instruction set is replaced with the new operation instruction sequence and then executed.

[0014] By adopting the above technical solution, the fixed operation instruction set is associated with and stored with reference execution context information when the instruction set is executed. Before execution, the system obtains the current execution context information and compares it with the reference execution context information; if they match, the system executes directly; if they do not match, a new sequence of operation instructions is regenerated based on the original natural language operation intent and the current execution context information. This solution enables the fixed operation instruction set to automatically adapt when the execution context conditions change, avoiding execution failures caused by context mismatch.

[0015] Optionally, the step of re-parseing the structured document data of the current webpage based on the original natural language operation intent and the current execution context information to generate a new sequence of operation instructions adapted to the current execution context information includes a re-matching step: The current execution context information is compared with the reference execution context information to extract condition deviation information; Obtain operation instruction information associated with the condition deviation information from the solidified operation instruction set. The operation instruction information includes at least one of the following: the source path of dynamic data, the location information of the operation target, and the operation type; wherein, the value of the dynamic data also changes with the execution context information. The original natural language operation intent, the conditional bias information, and the structured document data of the current webpage are input into the semantic understanding model. The semantic understanding model then uses the original natural language operation intent and the conditional bias information to re-match new operation instruction information corresponding to the operation instruction information in the structured document data of the current webpage. The original operation instruction information in the solidified operation instruction set is replaced with the new operation instruction information to update the solidified operation instruction set.

[0016] By adopting the above technical solution, in the re-matching step, the system compares the current execution context information with the reference execution context information, extracts the conditional deviation information between the two, and obtains the operation instruction information associated with the conditional deviation information from the fixed operation instruction set (including at least one of the following: the source path of dynamic data, the location information of the operation target, and the operation type). The original natural language operation intent, the conditional deviation information, and the structured document data of the current webpage are jointly input into the semantic understanding model. The semantic understanding model then re-matches new operation instruction information based on the original natural language operation intent and the conditional deviation information, replacing the original information. This solution makes the re-matching process targeted. Only the operation instruction information related to the conditional deviation is adjusted, rather than re-parsing all operation instructions, thereby improving the efficiency of re-matching and reducing the computational load of the semantic understanding model.

[0017] Optionally, the method further includes: During the execution of the operation instruction sequence, the execution context information corresponding to the operation instruction information is obtained, and a mapping relationship is established between different operation instruction information corresponding to different execution context information; the mapping relationship is stored in the fixed operation instruction set. The step of re-parses the structured document data of the current webpage based on the original natural language operation intent and the current execution context information to generate a new sequence of operation instructions adapted to the current execution context information further includes: The mapping relationship is queried first based on the current execution context information. If there is a mapping record in the mapping relationship that matches the current execution context information, the fixed operation instruction set is updated directly based on the operation instruction information corresponding to the mapping record. If there is no mapping record in the mapping relationship that matches the current execution context information, the re-matching step is performed, and the correspondence between the execution context information and the operation instruction information generated in this re-matching step is added to the mapping relationship as a new mapping record.

[0018] By adopting the above technical solution, during the execution of the operation instruction sequence, the system obtains the execution context information corresponding to the operation instruction information, establishes a mapping relationship between different operation instruction information corresponding to different execution context information, and stores this mapping relationship in the fixed operation instruction set. Before re-parsing, the system first queries this mapping relationship based on the current execution context information; if a matching mapping record exists, the fixed operation instruction set is directly updated based on the record, without needing to call the semantic understanding model for re-parsing; if no matching record exists, a re-matching step is performed, and the corresponding relationship generated this time is added to the mapping relationship as a new mapping record. This solution, through the accumulation and reuse of historical mapping records, eliminates the need to call the semantic understanding model again when triggering adaptation under the same execution context conditions, thereby reducing the number of times the semantic understanding model is called and shortening the context adaptation time.

[0019] Secondly, this application provides a natural language-driven web operating system, including, The natural language receiving module is used to receive natural language operation instructions input by the user; The instruction parsing and execution module is used to respond to the natural language operation instruction by calling the semantic understanding model to parse the structured document data of the target webpage, generate an operation instruction sequence to operate on the target webpage, and execute the operation instruction sequence. An operation recording module is used to record operation information corresponding to each operation step in the operation instruction sequence during the execution of the operation instruction sequence; An operation solidification module is used to respond to a solidification command triggered by the user, encapsulate the recorded operation information into a solidified operation command set, and associate and store the solidified operation command set with the URL information of the target webpage; The operation invocation module is used to respond to the user's invocation command for a stored solidified operation instruction set, match the corresponding solidified operation instruction set according to the invocation command, and execute the operation instruction sequence recorded in the solidified operation instruction set.

[0020] Thirdly, this application provides a natural language-driven webpage operation device, including a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed as described in any of the first aspects.

[0021] Fourthly, this application provides a computer-readable storage medium storing a computer program that can be loaded by a processor and executed as described in any of the first aspects.

[0022] In summary, this application includes at least one of the following beneficial technical effects: In this application, users do not need programming skills and can drive web page operations solely through natural language. Upon first execution, the semantic understanding model automatically parses and generates a sequence of operation instructions. During execution, operation information is recorded and encapsulated into a fixed set of operation instructions. When the fixed set of operation instructions is called on the same web page subsequently, the stored sequence of operation instructions is executed directly without calling the semantic understanding model for parsing again. This reduces the number of times the semantic understanding model is called, lowers the execution time of a single operation, and improves the execution speed of repetitive operations to the millisecond level. Furthermore, when changes occur in the structured document data of the target webpage, causing the operation information in the fixed operation instruction set to fail to locate the corresponding operation target, the system extracts the original natural language operation intent corresponding to the fixed operation instruction set, calls the semantic understanding model to re-parse the changed webpage, generates a new operation instruction sequence, and replaces the original operation information. This solution allows the fixed operation instruction set to continue to be used after the webpage structure changes, without requiring the user to re-execute the initial operation process, thereby reducing the maintenance cost of the fixed operation instruction set and extending its effective usage period. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1This is a flowchart illustrating a natural language-driven webpage operation method disclosed in an embodiment of this application.

[0025] Figure 2 This is a structural block diagram of a natural language-driven web operating system disclosed in an embodiment of this application.

[0026] Explanation of reference numerals in the attached diagram: 201, Natural Language Receiving Module; 202, Instruction Parsing and Execution Module; 203, Operation Recording Module; 204, Operation Consolidation Module; 205, Operation Invocation Module. Detailed Implementation

[0027] The following is in conjunction with the appendix Figure 1-2 This application will be described in further detail.

[0028] This application discloses a natural language-driven webpage operation method (hereinafter referred to as the operation method), the execution subject of which is the operating system. The following will combine... Figure 1 It elaborates on the detailed execution process of the operating system for the operation method.

[0029] S101 receives natural language operation instructions input by the user; S102, in response to natural language operation instructions, invokes the semantic understanding model to parse the structured document data of the target webpage, generates a sequence of operation instructions to operate on the target webpage, and executes the sequence of operation instructions; S103, during the execution of the operation instruction sequence, record the operation information corresponding to each operation step in the operation instruction sequence; S104, in response to the user-triggered fixed command, encapsulate the recorded operation information into a fixed operation command set, and associate and store the fixed operation command set with the URL information of the target webpage; S105, in response to the user's call command for the stored solidified operation instruction set, matches the corresponding solidified operation instruction set according to the call command and executes the operation instruction sequence recorded in the solidified operation instruction set.

[0030] S106, when executing the operation instruction sequence recorded in the solidified operation instruction set, if the operation information recorded in the solidified operation instruction set cannot locate the corresponding operation target in the structured document data of the current webpage, then extract the original natural language operation intent corresponding to the solidified operation instruction set, and based on the original natural language operation intent, call the semantic understanding model to re-parse the structured document data of the current webpage, generate a new operation instruction sequence, and replace the operation information recorded in the solidified operation instruction set with the new operation instruction sequence.

[0031] In implementation, users input natural language operation commands through the human-computer interaction interface of the device where the operating system is located. These natural language operation commands refer to the user's intended action on the target webpage, described in everyday natural language, such as "Submit a project application form on the XX approval system." After receiving the natural language operation command, the operating system temporarily stores it in memory and performs preliminary text preprocessing, including but not limited to removing extra spaces and standardizing punctuation, to obtain a standardized natural language operation command text.

[0032] After obtaining the standardized natural language operation instructions, the operating system accesses the target webpage and retrieves its complete structured document data. This structured document data refers to the target webpage's Document Object Model (DOM) tree, which uses a tree-like data structure to fully represent all elements on the target webpage, their hierarchical relationships, attribute information, and text content. The operating system accesses the target webpage by calling the browser engine built into the operating system or through a browser driver (such as Selenium WebDriver). Once the target webpage is loaded, it reads the complete DOM tree data of the target webpage through the Application Programming Interface (API) provided by the browser engine.

[0033] After reading the DOM tree data, the operating system does not directly input the complete DOM tree into the semantic understanding model. Instead, it preprocesses the DOM tree, extracting all interactive elements to generate a DOM summary table. The specific method for generating the DOM summary table is as follows: the operating system traverses all nodes in the DOM tree, filtering out interactive element nodes, which include, but are not limited to, input boxes. <input> Labels), buttons <button>Tags), drop-down selection box <select> Labels), text boxes< / select> <textarea> Tags), Links< / textarea> Tags) and block-level elements with click event listeners (such as tags) and block-level elements with click event listeners (such as tags) or The element must be a tag containing the `onclick` attribute or the `role="button"` attribute. For each selected interactive element node, the operating system extracts the following attribute information to generate a summary record: the element's tag type (e.g., "input", "button", "select"); the visible text or tag text displayed on the webpage (e.g., "project name", "submit"); the element's unique identifier (ID attribute value, e.g., "project-name"); the CSS selector automatically generated by the operating system based on the element's hierarchy and attribute information in the DOM tree (e.g., "#project-name" or "form#application-form>div.field-group>input#project-name"); the XPath expression automatically generated by the operating system based on the element's complete path from the DOM tree root node to the element (e.g., " / / input[@id='project-name']"); the element's current state information (including but not limited to visible / hidden, enabled / disabled, interactive / non-interactive); and the element's parent container information in the DOM tree (e.g., the identifier of the form area, the identifier of the navigation bar). The operating system organizes the summary records of all the interactive elements into a structured DOM summary table. This DOM summary table is organized in JSON array format or key-value pair table format, with each element node corresponding to an entry in the array, and each entry containing all the aforementioned attribute fields. By generating a DOM summary table instead of directly inputting the complete DOM tree, the operating system can effectively control the number of tokens input to the semantic understanding model, avoiding the model input exceeding the context window limit due to an excessively large DOM tree, while also improving the accuracy of the semantic understanding model in parsing interactive elements.

[0034] The semantic understanding model refers to a pre-trained Large Language Model (LLM), which possesses the ability to perform semantic understanding and reasoning on natural language text. The training process of the semantic understanding model includes: acquiring a training dataset containing webpage structured document data samples, corresponding natural language operation instruction samples, and labeled operation instruction sequence samples; using the webpage structured document data samples and natural language operation instruction samples as input and the labeled operation instruction sequence samples as output; and supervising and fine-tuning the basic language model to enable it to learn to generate corresponding operation instruction sequences based on the webpage structured document data and natural language operation intentions. When the operating system calls the semantic understanding model, it calls the application programming interface of the semantic understanding model deployed on a local server or cloud server, combining the DOM summary table and the pre-processed natural language operation instruction text into a prompt word conforming to the model input format, and sending it to the semantic understanding model. The semantic understanding model then performs semantic matching and reasoning to generate an operation instruction sequence for operating the target webpage.

[0035] The operation instruction sequence refers to a structured data set containing one or more operation steps arranged in execution order. Each operation step corresponds to an operation action in the user's intent, and each operation step includes an operation type field and a target element location information field. The specific reasoning process of the semantic understanding model is as follows: The semantic understanding model first parses the semantic information in the natural language operation instruction text to identify the operation target in the user's intent (such as "submit application form" or "query order"). The operation target refers to the task that the user expects to complete. Subsequently, the semantic understanding model performs semantic matching between the parsed operation target and the visible text, tag text, element type, and other attribute information of each interactive element in the DOM summary table to determine which elements are related to the operation target in the user's intent. Specifically, the semantic understanding model calculates the semantic similarity between keywords in the user's intent (such as "project name", "application amount", "submit") and the visible text, tag text, ID attribute value, and name attribute value of each interactive element, and filters interactive elements with semantic similarity exceeding a preset threshold as candidate operation target elements related to the user's intent.

[0036] For each candidate operation target element identified as related to user intent, the semantic understanding model further determines the type of operation that needs to be performed on that element. Specifically, the semantic understanding model determines the operation type based on a combination of the candidate operation target element's label type and visible text content: if the candidate operation target element is an input box type (i.e., ... <input> Tags that are not "button", "submit", "reset", or "hidden" in type 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19 ... <textarea> If the element is a dropdown selection box (i.e., a label), then it is determined that an input operation needs to be performed on the element; if the candidate operation target element is a dropdown selection box type (i.e., a label), then it is determined that an input operation needs to be performed on the element.< / textarea> <select> If the target element is a button (i.e., a button with the `type` attribute set to "button", "submit", or "reset"), then the element is considered to require a selection operation.< / select> <input> If the target element is a tag, or an interactive element with a click event listener, then the element is determined to require a click operation; if the candidate target element is a link (i.e., ...), then the element is considered to require a click operation. If an element has a tag and contains an href attribute, then it is determined that the element needs to perform a navigation operation.

[0037] When a semantic understanding model generates a complete sequence of operation instructions, it needs to determine the execution order of each operation step. The operating system uses the following rules to determine the execution order: First rule: Determined by the order of the form structure. The operating system checks whether the candidate operation target elements belong to the same form container (i.e., have the same parent). <form> (element). If multiple candidate operation target elements belong to the same form container, the operating system arranges their corresponding operation steps according to the visual arrangement order of these elements from top to bottom and from left to right within the form container (i.e., the document position order of each element in the DOM tree, determined by comparing the document position relationship returned by the compareDocumentPosition method of each element).

[0038] The second rule: Priority order based on operation type. For candidate operation target elements that do not belong to the same form container, the operating system determines the execution order according to the preset priority of the operation type. The preset priority, from high to low, is as follows: input operations and selection operations take precedence over click operations, and click operations take precedence over navigation operations. For example, when the candidate operation target elements contain both input box elements and button elements, the operating system will execute the input operation steps corresponding to the input box before the click operation steps corresponding to the button; when the candidate operation target elements contain both button elements and link elements, the operating system will execute the click operation steps corresponding to the button before the navigation operation steps corresponding to the link.

[0039] The third rule: Determined based on the sequence number information in the element identifier. The operating system checks whether the ID attribute value, name attribute value, or visible text of the candidate operation target element contains a numeric sequence number (such as "Name1", "Name2", "Name3", or "row_1", "row_2", "row_3"). If a numeric sequence number is detected, the operating system arranges the corresponding operation steps in ascending order of the numeric sequence number (smaller sequence numbers are executed first, and larger sequence numbers are executed last).

[0040] Fourth rule: Determined based on semantic dependencies. The operating system detects whether the visible text or attribute values ​​of candidate operation target elements contain semantic keywords indicating dependencies. These semantic keywords include, but are not limited to: "confirm" depending on "submit", "next step" depending on "previous step", and "save" depending on "fill in". When the operating system detects the above semantic dependencies between the visible text or attribute values ​​of two candidate operation target elements, the operating system will execute the operation steps corresponding to the dependent party after the operation steps corresponding to the dependent party.

[0041] The operating system applies the rules sequentially according to their priority (i.e., the first rule is applied first, and if the first rule cannot uniquely determine the order, the second rule is applied, and so on), until the execution order of the operation steps for all candidate operation target elements is determined. If, after applying all four rules, the operating system still finds that the order of operation steps for some candidate operation target elements cannot be uniquely determined (i.e., two or more candidate operation target elements have the same priority under all rules), the operating system arranges their corresponding operation steps according to the order in which these indistinguishable candidate operation target elements appear in the DOM summary table (i.e., the order in which interactive elements are traversed when the operating system traverses the DOM tree and generates the DOM summary table). The operating system then arranges all operation steps with the final determined execution order in chronological order, assigning each operation step an incrementing number starting from 1, forming a complete sequence of operation instructions.

[0042] Taking the user input "Submit Project Application Form" as an example, the semantic understanding model matches three candidate operation target elements: an input box with visible text "Project Name", an input box with visible text "Application Amount", and a button with visible text "Submit". The operating system detects that the three belong to the same form container. According to the first rule, based on their visual arrangement in the form container (input box on top, button on the bottom), the execution order is determined as follows: first, the input operation of the "Project Name" input box is executed; then, the input operation of the "Application Amount" input box is executed; and finally, the click operation of the "Submit" button is executed, thus generating a complete sequence of operation instructions. The sequence of operation instructions exists in a structured data format, such as being stored in the operating system's memory as a JSON array, with each array element representing an operation step.

[0043] After obtaining the sequence of operation instructions, the operating system iterates through each operation step in the sequence and identifies whether the operation step requires operation parameters. Specifically: if the operation type of the operation step is "input" or "select", then the operation step requires operation parameters (i.e., the text content to be filled in or the option value to be selected); if the operation type of the operation step is "click", "scroll", "wait", or "navigate", then the operation step does not require operation parameters.

[0044] For operation steps that require parameters, the operating system extracts the corresponding parameters from the original natural language operation instruction text. Specifically, the operating system obtains the visible text tag or ID attribute value (such as "project name" or "application amount") of the target element of the operation step as keywords, and searches for data content semantically matching these keywords in the original natural language operation instruction text. For example, if the original natural language operation instruction text is "Submit project application form, project name is smart city construction project, application amount is 5 million yuan", the operating system matches "smart city construction project" as the keyword and "5 million yuan" as the keyword in the instruction text. Then, the operating system uses "smart city construction project" as the operation parameter for this operation step and "5 million yuan" as the operation parameter for the corresponding operation step.

[0045] If the original natural language operation instruction text does not contain data content matching the keyword (for example, the user only enters "submit project application form" without providing the project name and the specific amount to be applied for), the operating system determines that the operation step is missing operation parameters. For all operation steps with missing operation parameters, the operating system first initiates a query with the user through an interactive dialogue before executing the subsequent operation to obtain the missing parameter information item by item. Specifically, the operating system generates a rhetorical question containing the keyword corresponding to the missing parameter (such as "please provide the project name" or "please provide the amount to be applied for"), and displays it to the user through the user interface. After each rhetorical question, the user supplements the corresponding data content through a dialog box or voice. The operating system receives the supplementary data input by the user and uses it as the operation parameter for that operation step. After collecting all the complete operation parameters required for all operation steps in the above manner, the operating system fills each operation parameter into the operation parameter field in the instruction data of the corresponding operation step, forming a complete sequence of operation instructions.

[0046] After obtaining a complete sequence of operation instructions containing all operation parameters, the operating system executes each operation step sequentially according to its step number. When executing each operation step, the operating system first searches for the corresponding target element in the DOM tree of the current target webpage based on the target element location information for that operation step. If the target element location information contains multiple location methods, the operating system attempts to use each method in a preset priority order: first, it tries to use CSS selectors; if that fails, it tries XPath expressions; if that also fails, it tries element ID attribute values; finally, it tries fuzzy matching of the element's visible text tags in the DOM tree. If any location method successfully finds the corresponding target element, the operating system stops trying subsequent location methods and uses the successfully located element as the target of the current step. If the target element location information for that step contains only one location method and the location fails, the operating system continuously checks for the target element every 200 milliseconds according to a preset timeout period (default 3000 milliseconds) until the timeout expires or the target element is successfully located. If the target element is successfully located within the timeout period, the operating system performs the corresponding operation on the target element according to the operation type field of this step: If the operation type is "input", the operating system fills the target element (usually an input box or text box) with the text content stored in the operation parameter field; if the operation type is "click", the operating system triggers the click event of the target element (usually a button or link); if the operation type is "select", the operating system sets the selected item of the target element (usually a drop-down selection box) to the option value stored in the operation parameter field; if the operation type is "scroll", the operating system scrolls the target element into the browser's visible area; if the operation type is "wait", the operating system pauses execution and waits for the duration specified in the operation parameter field; if the operation type is "navigate", the operating system navigates the current browser window to the Uniform Resource Locator (URL) specified in the operation parameter field.

[0047] During the execution of each operation step, the operating system synchronously monitors the execution process of that step to determine whether the step was successful or failed. The criteria for determining "step execution failure" include, but are not limited to, the following: Scenario 1: The operating system locates the target element in the current webpage DOM tree based on the target element location information. If the corresponding target element cannot be located within the timeout period (default 3000 milliseconds, set by the timeout field), the operation step is determined to have failed due to "element location timeout". Scenario 2: If the operating system successfully locates the target element, but the target element is in a non-interactive state (e.g., the "disabled" property of the target element is true, the "visible" property of the element is false, or the width and height of the element are 0), then the operation step is determined to fail because the "target element is not interactive". Scenario 3: The operating system successfully locates the target element and the target element is interactive, but an exception occurs during the operation (such as the element being obscured by other elements during the click operation, preventing the click event from being triggered, or the element's read-only property being true during the input operation, preventing the input content from being written). In this case, the operation step is determined to have failed due to "operation execution exception". The timeout and retry count fields are used to handle the above-mentioned failure scenarios: When the operating system executes each operation step, it first reads the timeout field of that step (default value is 3000 milliseconds), and checks whether the target element appears within the time window specified by the field, checking once every 200 milliseconds. If the target element is successfully located within the timeout window, the loop stops and execution continues. If the target element is not located after the timeout, an "element location timeout" error is recorded. After the operating system detects that the step execution has failed, it reads the retry count field of that step (default value is 3 times), and repeats the entire process of that step (from locating the target element to executing the operation) within the number of times specified by the field. Each retry is spaced 500 milliseconds apart. If the step still fails after the number of retries is exhausted, the operating system determines that the operation step has ultimately failed, suspends the execution of subsequent steps, and enters the fault handling process.

[0048] In the fault handling process, the operating system performs the following operations: First, the operating system records the complete information of the currently failed operation step, including the step number, operation type, target element location information, operation parameters, failure reason type (one of "element location timeout", "target element is not interactive", "operation execution exception"), and a detailed error description at the time of failure (such as "CSS selector #project-name did not match any element" or "click operation was obscured by element #overlay"). The above information is combined into a failure record and stored in the operating system's memory cache.

[0049] Secondly, the operating system determines whether the current failed step belongs to the execution process of a fixed operation instruction set (i.e., the execution triggered by the user calling a stored fixed operation instruction set) or to the first execution process (i.e., the execution triggered by the user inputting natural language operation instructions). If the current failed step belongs to the first execution process, the operating system displays a failure message on the user interface, informing the user that "Operation execution failed, the reason for failure is: [specific error description], please check the webpage status or adjust the operation instructions and try again", and terminates the current operation process, without entering the subsequent fixed operation instruction set encapsulation process. If the current failed step belongs to the execution process of a fixed operation instruction set, the operating system further determines whether the fixed operation instruction set has undergone automatic repair (i.e., whether the version number of the instruction set is greater than 1, or whether the instruction set contains a repair history). If the fixed operation instruction set has never undergone automatic repair, the operating system triggers an automatic repair process for the fixed operation instruction set (the specific content of the repair process is described in the following description of step S106). If the fixed operation instruction set has undergone automatic repair but still fails to execute, the operating system determines that automatic repair cannot solve the current problem, abandons further repair, presents a failure message on the user interface, informing the user that "the fixed operation instruction set has failed to execute, automatic repair cannot be restored, and it is recommended that the user re-execute the operation to overwrite the original fixed operation instruction set," and marks the fixed operation instruction set as "invalid," while retaining its historical operation information sequence and failure record for auditing and reference.

[0050] After the operating system completes the execution of the entire sequence of operation instructions (i.e., all operation steps are successfully executed) or terminates execution (i.e., subsequent steps are paused due to the failure of a certain operation step), it arranges all the operation information recorded during this execution process (including records of successfully executed operation steps and failure records of failed steps) in the execution order to form a complete execution record of this operation, and persists this complete execution record to the operating system's local database or associated cloud storage service.

[0051] The operation information refers to the recorded data generated during the execution of each operation step, used to describe the execution status of that operation step. Its content corresponds to the instruction data of that operation step in the operation instruction sequence, but further includes an execution timestamp and execution result status. Specifically, after each operation step is executed, the operating system immediately combines the step number, operation type, target element location information (including at least one of CSS selectors, XPath expressions, element ID attribute values, and element visible text tags), operation parameters, execution timestamp (execution completion time accurate to milliseconds), execution result status ("success" or "failure"), and a description of the error reason (if any) for failure into an operation information record, and appends this operation information record to the operating system's memory cache. After all operation steps are executed (whether all are successful or terminated midway due to failure), the operating system arranges all operation information records stored in the memory cache in execution order to form a complete execution record for this operation, and persistently stores this complete execution record in the operating system's local database or an associated cloud storage service. In addition, the complete execution record also stores the following information: the original text content of the natural language operation instructions entered by the user, the complete Uniform Resource Locator (URL) of the target webpage, and the execution time range of this operation (start time and end time). During the execution process, even if an operation step fails, the recorded historical operation information is still retained in the cache and will not be discarded or overwritten due to execution failure.

[0052] After completing the persistent storage, the operating system presents the execution result of this operation to the user on the user interface, including a status message indicating whether the operation was successful or failed, as well as a summary of the execution steps.

[0053] While or after presenting the execution results to the user, the operating system provides an interactive entry point on the user interface for triggering the solidified operation (e.g., a button labeled "Save as Skill" or "Solidify this operation"). When the user needs to save the operation process as a reusable solidified operation instruction set, the user triggers the solidification instruction by clicking or touching this interactive entry point. In response to the user-triggered solidification instruction, the operating system first displays a naming dialog box on the user interface, allowing the user to enter a name for the solidified operation instruction set. After receiving the name entered by the user, the operating system checks whether the name is the same as an existing solidified operation instruction set stored in the operating system that belongs to the user. If a name conflict exists, the operating system displays a conflict warning dialog box on the user interface, providing the user with the following options: overwrite the existing solidified operation instruction set, rename the current solidified operation instruction set, or cancel the current solidification operation. The operating system executes the corresponding operation based on the user's selection. If the user chooses to overwrite or the name entered by the user does not conflict with any existing solidified operation instruction set, the operating system enters the solidified operation instruction set encapsulation process.

[0054] In the encapsulation process of the solidified operation instruction set, the operating system extracts operation information (including operation type, target element location information, operation parameters, etc.) for each operation step from the complete execution record of this operation, and arranges the extracted operation information in the execution order to form an operation information sequence. Simultaneously, the operating system stores the text content of the user-inputted raw natural language operation instructions as the raw natural language operation intent. The operating system then encapsulates the above operation information sequence, the raw natural language operation intent, the user-specified solidified operation instruction set name, and the target webpage's Uniform Resource Locator (URL) into an independent executable solidified operation instruction set according to a preset data structure. The solidified operation instruction set exists in the form of a structured data file (e.g., stored in JSON or Extensible Markup Language (XML) format on the operating system's local disk or associated cloud storage). Its data structure includes the following fields: a unique identifier for the solidified operation instruction set (a unique number automatically generated by the operating system, such as "sk_20260115_001"); the user-specified solidified operation instruction set name; the original natural language operation intent (i.e., the natural language instruction text initially entered by the user); the target webpage's URL information, specifically including the complete URL, domain name, and first-level path prefix; the operation information sequence (i.e., the operation information of each operation step arranged in execution order); the version number (initially version 1); the creation timestamp; the update timestamp; and execution status statistics (including total number of calls, number of successful executions, the time of the most recent successful execution, and the time of the most recent failure). The operating system stores the encapsulated solidified operation instruction set in its solidified instruction repository and associates it with the target webpage's URL information for URL matching during subsequent calls.

[0055] The specific method for associating the solidified operation instruction set with the target webpage's URL information is as follows: The operating system extracts the domain name portion (e.g., "xxx.gov.cn") and the first-level path prefix portion (e.g., " / apply") from the complete URL of the target webpage, and uses the combination of the domain name and the first-level path prefix as the binding URL identifier for the solidified operation instruction set. When storing the solidified operation instruction set, the operating system writes this binding URL identifier into the associated storage field of the solidified operation instruction set. When a user subsequently calls the solidified operation instruction set, the operating system obtains the complete URL of the current webpage, extracts the domain name and the first-level path prefix of the current webpage, and compares them with the binding URL identifier stored in the solidified operation instruction set. If the comparison result is consistent (i.e., the domain name and the first-level path prefix are the same), the operating system determines that the current webpage matches the solidified operation instruction set and allows the call; if the comparison result is inconsistent, the operating system determines that the current webpage does not match the solidified operation instruction set, displays a message on the user interface stating "The current page does not match the website bound to the solidified operation instruction set, and the solidified operation instruction set cannot be called," and suggests that the user re-execute the operation to generate a new solidified operation instruction set.

[0056] After the operating system stores the firmware instruction set in the firmware instruction repository, the user can invoke the firmware instruction set when the same operation needs to be performed later. Specifically, the user can browse or search for the stored firmware instruction set through the firmware instruction set list or search box provided on the operating system's user interface, and initiate the invocation command by clicking or touching the invocation button on the corresponding entry.

[0057] In response to a user's call to a stored set of fixed operation instructions, the operating system first searches for and matches the corresponding fixed operation instruction set in the fixed instruction repository based on the name or unique identifier of the user-selected set. Upon successful matching, the operating system reads the operation information sequence of the fixed operation instruction set and executes each operation step sequentially according to the execution order of the operation steps in the operation information sequence. The specific execution method of each operation step is the same as the execution method of the operation instruction sequence described above: locating the target element in the DOM tree of the current webpage based on the target element location information in the operation information, and performing the corresponding operation (click, input, selection, scrolling, etc.) according to the operation type. When executing a fixed operation instruction set, the operating system executes directly based entirely on the operation information sequence stored within the fixed operation instruction set, without needing to call the semantic understanding model again to parse the structured document data of the current webpage, nor needing to generate the operation instruction sequence again. Thus, the operating system achieves a complete process from "natural language-driven initial execution" to "scripted subsequent reuse," significantly reducing the number of calls to the semantic understanding model and the consumption of computational resources when repeatedly executing the same operation, and reducing the operation execution time from several seconds for the first execution to milliseconds for subsequent executions.

[0058] If, during the execution of a sequence of operation information recorded in a fixed set of operation instructions by the operating system, the operating system is unable to locate the corresponding operation target in the DOM tree of the current webpage based on the target element location information in the operation information of that operation step, then the operating system determines that the fixed set of operation instructions has become invalid on the current webpage.

[0059] The phrase "unable to locate the corresponding operation target" specifically includes, but is not limited to, the following situations: the CSS selector recorded in the operation information does not match any element node in the current webpage DOM tree; the XPath expression recorded in the operation information does not match any element node in the current webpage DOM tree; the element ID attribute value recorded in the operation information cannot find a corresponding element in the current webpage DOM tree; the visible text tag of the element recorded in the operation information cannot find an element with that text content in the current webpage DOM tree; or although the target element is located, its state is non-interactive (e.g., the element is "hidden" or "disabled," and cannot be clicked or entered). When the operating system detects any of the above situations, the operating system suspends the execution of the current fixed operation instruction set, records the step number of the current failed step, the failure reason type ("element does not exist," "element is not interactive," "element is obscured," etc.), and the original operation information of the step, and enters the automatic repair process.

[0060] In the automatic repair process, the operating system first extracts the original natural language operation intent (i.e., the natural language instruction text entered by the user when first performing the operation) stored in the fixed operation instruction set, and extracts the complete structured document data of the current webpage (i.e., the DOM tree of the current webpage). It then performs the same preprocessing operations as before on the DOM tree of the current webpage to generate a DOM summary table for the current webpage. Next, the operating system combines the original natural language operation intent and the DOM summary table of the current webpage into a prompt word, calls the semantic understanding model, and the semantic understanding model re-parses the DOM summary table of the current webpage based on the original natural language operation intent, generating a new sequence of operation instructions adapted to the structure of the current webpage. After obtaining the new sequence of operation instructions returned by the semantic understanding model, the operating system replaces the original operation information sequence in the fixed operation instruction set with this new sequence, increments the version number of the fixed operation instruction set by 1, and updates the update timestamp. After the replacement is complete, the operating system uses the new sequence of operation instructions to continue executing subsequent operations from the failed step until all operation steps are completed. If the semantic understanding model cannot match enough interactive elements in the DOM summary table of the current webpage based on the original natural language operation intent, and cannot generate a complete sequence of operation instructions, the operating system determines that automatic repair has failed. It then pushes a repair failure message to the user interface, informing the user that "the current webpage structure has changed too much to be automatically repaired. Please re-execute the operation to overwrite the original fixed operation instruction set," and marks the fixed operation instruction set as "invalid," while retaining its historical operation information sequence for auditing and reference. After receiving this message, the user can generate a new fixed operation instruction set by re-executing the complete natural language-driven operation process (i.e., re-executing steps S101 to S104). The newly generated fixed operation instruction set automatically overwrites the original invalid instruction set.

[0061] Optionally, the operation method also includes: S301, during the execution of the operation instruction sequence, identify the dynamic data in the operation instruction sequence and obtain the source path of the dynamic data, wherein the source path is used to indicate the way to obtain the current value of the dynamic data when the fixed operation instruction set is executed; S302, in response to a user-triggered fixed instruction, associates the source path with the operation steps corresponding to the dynamic data and stores them in the fixed operation instruction set; S303, when executing the solidified operation instruction set, obtains the current value of dynamic data in real time according to the source path and executes the corresponding operation steps.

[0062] Specifically, the step of "obtaining the source path of dynamic data" in S301 is as follows: when dynamic data corresponds to multiple source paths, obtain the original natural language operation intent corresponding to the fixed operation instruction set, calculate the semantic matching degree between the original natural language operation intent and the data semantic tag associated with each source path, and take the source path with the highest semantic matching degree as the source path of dynamic data.

[0063] In implementation, the operating system synchronously identifies dynamic data within the sequence of operation instructions during execution. Specifically, the operating system iterates through each operation step in the sequence, checking whether the operation type is "input" or "select". If the operation type is either of these two, the operation step involves data entry or selection, and the operating system determines the corresponding data to be entered or the option to be selected as dynamic data. If the operation type is "click", "scroll", "wait", or "navigate", the operation step does not involve data entry or selection, and the operating system does not perform dynamic data identification for that operation step.

[0064] For each piece of dynamic data, the operating system obtains the source path of that dynamic data. The source path refers to structured descriptive information indicating the source of the dynamic data, specifically represented as a structured data description object containing a source type field and a source location information field. The source type indicates the data source category identified when the dynamic data is solidified, including two main categories: "instruction-based acquisition" and "webpage element reference." "Instruction-based acquisition" means that the specific value of the dynamic data originates from the parsed result of a natural language operation instruction or data provided by the user through interactive queries; "webpage element reference" means that the specific value of the dynamic data originates from the text content or attribute value of an interactive element in the current target webpage or other related webpages.

[0065] The specific methods by which the operating system obtains the source path of dynamic data are as follows: The operating system obtains the visible text label or ID attribute value of the target element corresponding to the operation step as a keyword, and searches for data content that semantically matches the keyword in the original natural language operation instruction text.

[0066] If the original natural language operation command text contains data content that matches the keyword (for example, if the original natural language operation command text is "Submit project application form, project name is smart city construction project, application amount is 5 million yuan", and the operating system matches "5 million yuan" with "application amount" as the keyword), then the operating system sets the source type of the source path of the dynamic data to "from command parsing", and stores the parsing rules for extracting the specific values ​​of the dynamic data from the natural language operation command text in the source location information field of the source path. The parsing rules include keyword matching rules and value extraction rules (such as "using 'application amount' as the keyword, extract the value immediately following it or the content within the quotation marks, or extract the content from the word 'for' to the next period").

[0067] If the original natural language operation command text does not contain data content matching the keyword (for example, the user only enters "submit project application form" without providing the project name and the specific value of the application amount), the operating system determines that the dynamic data cannot be obtained from the command, sets its source type to "from user interaction", and stores the name prompt text of the dynamic data in the source location information field of the source path, which is used to initiate an interactive query to the user to obtain the data during execution.

[0068] If the operating system does not match the data content corresponding to the dynamic data in the original natural language operation instruction text, it further searches for candidate source path elements of the dynamic data in the intermediate inference results of the semantic understanding model. The candidate source path elements refer to interactive elements that are semantically related to the operation step, identified by the semantic understanding model during the semantic matching process between the operation target and the DOM summary table. If only one candidate source path element is found, the operating system directly determines this candidate source path element as the source path element of the dynamic data. If multiple candidate source path elements are found, the operating system selects one as the final source path element from among them in the following way: the operating system obtains the original natural language operation intent corresponding to the fixed operation instruction set and generates corresponding data semantic tags for each candidate source path element. The data semantic tag refers to natural language text that describes the business meaning of the data provided by the candidate source path element. Its generation method is as follows: The operating system extracts at least one text information from the visible text, tag text, ID attribute value, name attribute value, placeholder attribute value, and title or area identifier of the interactive element corresponding to the candidate source path element. This extracted text information is input into a semantic understanding model, which extracts core business meaning keywords. These core business meaning keywords are then used as the data semantic tag of the candidate source path element. The operating system calculates the semantic matching degree between the original natural language operation intent and the data semantic tag of each candidate source path element. The candidate source path element with the highest semantic matching degree is selected as the source path element of the dynamic data. The source type of the dynamic data's source path is set to "based on webpage element reference." The data source location information corresponding to the source path element (including but not limited to at least one of CSS selectors, XPath expressions, element ID attribute values, and element visible text tags) is stored in the source location information field of the source path. If two or more candidate source path elements have the same semantic matching score and all of them are the highest value, the operating system pushes a candidate path selection prompt to the user interface, displays the data semantic labels of each candidate source path element and the visible text of the corresponding interactive elements, and allows the user to manually select the desired source path element. The operating system then stores the data source location information corresponding to the source path element selected by the user as the source path of the dynamic data.

[0069] The semantic matching score is calculated as follows: The operating system obtains the original natural language operation intent text corresponding to the fixed operation instruction set, and the data semantic label text of a candidate source path element. The operating system inputs the original natural language operation intent text and the data semantic label text into the semantic understanding model, which performs semantic encoding on the two texts respectively. Specifically, the semantic understanding model, based on its internal embedding layer and encoder structure, transforms the input natural language text into a fixed-dimensional semantic vector. This process utilizes the language representation capabilities learned by the semantic understanding model during the pre-training stage, which is the same semantic understanding mechanism used by the aforementioned semantic understanding model to parse natural language operation instructions and generate operation instruction sequences, both of which transform text into semantic vector representations in a high-dimensional space. After the semantic understanding model completes the vector encoding of the two texts, the operating system calls the cosine similarity calculation function built into the semantic understanding model to calculate the cosine value of the angle between the two semantic vectors, and outputs a value between 0 and 1 as the semantic matching score of the candidate source path element. The cosine similarity calculation function is a basic mathematical operation capability that the semantic understanding model already possesses during its pre-training stage. The closer the numerical value calculated by the semantic understanding model is to 1, the closer the data semantic label of the candidate source path element is to the vector direction of the original natural language operation intent in the high-dimensional semantic space, that is, the more semantically matched the two are; the closer the numerical value is to 0, the closer the two vectors are to approximately orthogonal in the high-dimensional semantic space, that is, the less semantically matched the two are.

[0070] Taking the user input "Submit Project Application Form" as an example, the original natural language operation command text does not contain the specific value of "Application Amount". The operating system cannot obtain the source of this dynamic data from the command. Therefore, the operating system obtains two candidate source path elements in the intermediate inference results of the semantic understanding model: an input box with visible text "Project Approval Amount" and an input box with visible text "Approval Amount". The operating system generates data semantic labels "Project Approval Amount" and "Approval Amount" for these two candidate source path elements respectively, obtains the original natural language operation intent "Submit Project Application Form", and calculates the semantic matching score between "Submit Project Application Form" and "Project Approval Amount" and between "Submit Project Application Form" and "Approval Amount". The operating system determines the candidate source path element with the higher score as the source path element of the dynamic data and stores its corresponding data source location information as the source path of the dynamic data. If the semantic understanding model only matches one interactive element in the intermediate inference results that has a semantic association with the operation step, the operating system directly determines that interactive element as the source path element of the dynamic data and does not execute the above semantic matching calculation process.

[0071] After determining the source path of dynamic data, the operating system associates the source path with the corresponding operation steps in memory. Subsequently, in response to a user-triggered fixed instruction, the operating system executes the encapsulation process of the fixed operation instruction set. In this encapsulation process, the operating system reads each piece of dynamic data and its associated stored source path from memory, and encapsulates the source path as part of the operation information for the corresponding operation step, along with the operation type and target element location information, into the fixed operation instruction set according to a preset data structure. Specifically, for the operation steps corresponding to dynamic data, the operation parameter fields in the operation information do not store specific values, but rather the source path of the dynamic data, i.e., the source type field ("obtained based on instruction" or "based on webpage element reference") and the source location information field (parsing rules, name prompt text, or data source location information).

[0072] After the operating system stores the firmware instruction set in the firmware instruction repository, when a user invokes the firmware instruction set, the operating system executes each operation step sequentially according to the execution order of the operation information sequence in the firmware instruction set. When executing a certain operation step, the operating system first checks whether the operation information for that step contains a source path for the dynamic data. If the operation step does not contain a source path, the operating system directly executes the step according to the aforementioned operation instruction sequence. If the operation step contains a source path, the operating system reads the source type field of the source path and performs the corresponding data acquisition operation based on the source type. If the source type is "from instruction parsing", the operating system reads the parsing rules stored in the source location information field of the source path and checks whether the current call is accompanied by a new natural language description (i.e., whether the user provides new natural language operation instruction text in the instruction input box on the call interface when calling the fixed operation instruction set). If the current call is accompanied by a new natural language description, the operating system obtains the new natural language operation instruction text, re-parses and extracts the current specific value of the dynamic data from the new natural language operation instruction text according to the parsing rules, and uses the parsed value as the current value of the dynamic data; if the current call is not accompanied by a new natural language description, the operating system uses the original natural language operation intent text stored during the fixed process, parses and extracts the current specific value of the dynamic data from the original natural language operation intent text according to the parsing rules, and uses the parsed value as the current value of the dynamic data.

[0073] If the source type is "from user interaction", the operating system will pop up an input dialog box on the user interaction interface, displaying the name prompt text stored in the source location information field (such as "Please provide the declared amount"), and wait for the user to supplement the data through dialog box input or voice input, and use the data entered by the user as the current value of the dynamic data.

[0074] If the source type is "based on webpage element reference", the operating system reads the data source location information from the source path's source location information field and locates the corresponding data source element in the DOM tree of the current webpage based on this information. If the location is successful in the DOM tree, the operating system extracts the text content or value attribute of the data source element, uses the extracted content as the current value of the dynamic data, and performs data type conversion on the extracted text content. If the target input box's type attribute is "number" or the input box expects to receive numeric data, the operating system converts the extracted text content from string to numeric before proceeding with the subsequent data entry operation. If the extracted text content contains currency symbols or thousands separators, the operating system removes these symbols before performing data type conversion, retaining only numeric characters and decimal points. If the location fails to be found in the DOM tree based on the data source location information, the operating system enters an automatic repair process and re-matches the corresponding data source for the dynamic data in the current webpage.

[0075] After obtaining the current value of the dynamic data, the operating system fills the current value into the target element of the current operation step, thus completing the execution of that operation step. The operating system executes all operation steps in the fixed operation instruction set in the above manner until all operation steps are completed or an operation step fails and enters the fault handling process.

[0076] Optionally, the fixed operation instruction set also stores reference execution context information when the operation instruction sequence is executed; The step of executing the sequence of operation instructions recorded in the solidified operation instruction set in S105 includes the following sub-steps: S1051, Obtain the current execution context information and the reference execution context information corresponding to the fixed operation instruction set; Verify whether the current execution context information matches the reference execution context information; S1052, If a match is found, execute the sequence of operation instructions recorded in the solidified operation instruction set; S1053, if there is no match, obtain the original natural language operation intent corresponding to the solidified operation instruction set, and based on the original natural language operation intent and the current execution context information, re-parse the structured document data of the current webpage, generate a new operation instruction sequence adapted to the current execution context information, and then replace the corresponding operation instruction sequence in the solidified operation instruction set with the new operation instruction sequence before execution.

[0077] S401, during the execution of the operation instruction sequence, obtain the execution context information corresponding to the operation instruction information, and establish a mapping relationship between different operation instruction information corresponding to different execution context information; store the mapping relationship in the fixed operation instruction set; The step S1053, "based on the original natural language operation intent and the current execution context information, re-parses the structured document data of the current webpage to generate a new sequence of operation instructions adapted to the current execution context information," specifically includes the following steps: First, query the mapping relationship based on the current execution context information. If there is a mapping record in the mapping relationship that matches the current execution context information, then directly update the fixed operation instruction set according to the operation instruction information corresponding to the mapping record. If there is no mapping record in the mapping relationship that matches the current execution context information, then perform the re-matching step, and add the correspondence between the execution context information and the operation instruction information generated in this re-matching step as a new mapping record to the mapping relationship. The rematching process includes: The current execution context information is compared with the reference execution context information to extract condition deviation information; Obtain operation instruction information associated with condition deviation information from the fixed operation instruction set. The operation instruction information includes at least one of the following: the source path of dynamic data, the location information of the operation target, and the operation type; wherein, the value of the dynamic data also changes with the execution context information. The original natural language operation intent, conditional bias information, and the structured document data of the current webpage are input into the semantic understanding model. Based on the original natural language operation intent and conditional bias information, the semantic understanding model re-matches the new operation instruction information corresponding to the operation instruction information in the structured document data of the current webpage. The original operation instruction information in the fixed operation instruction set is replaced with the new operation instruction information to update the fixed operation instruction set.

[0078] In implementation, when the operating system invokes a fixed operation instruction set for execution, it first obtains the current execution context information. This execution context information refers to the environmental state information at the time of execution of the fixed operation instruction set, including but not limited to: the Uniform Resource Locator (URL) of the current webpage, the current user's identity and permission information, the current time information, the existence status of specific elements on the current webpage, and the completion status of the preceding operations. Simultaneously, the operating system obtains reference execution context information pre-associated and stored within the fixed operation instruction set. This reference execution context information refers to the execution context information synchronously collected by the operating system when the fixed operation instruction set is first generated (i.e., when the user triggers the fixed instruction), and when the corresponding operation instruction sequence is executed. The operating system compares the current execution context information with the reference execution context information to verify their match. If they match, the operating system directly executes the operation instruction sequence recorded in the fixed operation instruction set; if they do not match, the operating system enters the context adaptation process.

[0079] Specifically, the operating system compares the current execution context information with the reference execution context information item by item to verify whether they match. The matching verification method is as follows: the operating system compares each environmental state parameter in the current execution context information with the corresponding environmental state parameters in the reference execution context information. For numerical parameters (such as timestamps, webpage loading progress percentages, etc.), the operating system calculates the absolute value of the difference between the current value and the reference value. If this absolute value is less than or equal to a preset tolerance threshold (e.g., time difference less than or equal to 5 minutes, loading progress difference less than or equal to 5%), the parameter is considered a match; if the absolute value is greater than the preset tolerance threshold, the parameter is considered a mismatch. For categorical parameters (such as user roles, the domain name portion of a webpage URL, etc.), the operating system determines whether the current value and the reference value are exactly the same. If they are exactly the same, the parameter is considered a match; if they are not the same, the parameter is considered a mismatch.

[0080] After comparing each environment state parameter item by item, the operating system determines that the current execution context information matches the reference execution context information as a whole if all parameters match. If one or more parameters do not match, the operating system determines that the current execution context information does not match the reference execution context information as a whole and enters the context adaptation process. For example, if the reference execution context information records that the user role when executing the fixed operation instruction set is "administrator", while the current user role in the current execution context information is "ordinary user", the operating system determines that this parameter item does not match, and thus determines that the current execution context information does not match the reference execution context information as a whole, and enters the context adaptation process. If the reference execution context information records the execution time as "2026-06-24 10:00:00", while the current time in the current execution context information is "2026-06-24 10:03:00", the time difference between the two is 3 minutes, which is less than or equal to the preset tolerance threshold of 5 minutes, then the operating system determines that this time parameter item matches.

[0081] When the operating system determines that the current execution context information does not match the reference execution context information, the operating system enters the context adaptation process. In this process, the operating system first queries the mapping table stored in the fixed operation instruction set based on the current execution context information (the generation method of the mapping table is detailed below) to determine if there is a mapping record in the table that matches the current execution context information. If a matching mapping record exists, the operating system directly updates the fixed operation instruction set based on the corresponding operation instruction information and then continues execution using the updated instruction information, without needing to call the semantic understanding model for re-parsing. If no matching mapping record exists in the table, the operating system performs a re-matching step.

[0082] The semantic understanding model utilizes the current execution context information in the following way: The semantic understanding model parses the various environmental state parameters in the current execution context information and identifies the constraint parameters related to webpage operations. The constraint parameters include, but are not limited to, the current user's role identifier (used to determine the range of elements that the user can operate, for example, an administrator role can operate all elements, while a regular user role can only operate some elements), the current time information (used to determine the input values ​​of time-sensitive data involved in the operation steps, for example, if the operation includes a "enter today's date" step, the semantic understanding model generates the corresponding date value based on the current time information), and the current webpage loading status (used to determine whether a waiting step needs to be inserted in the operation steps).

[0083] When matching interactive elements, the semantic understanding model filters candidate elements in the DOM summary table based on the aforementioned constraint parameters. For example, when the current execution context indicates that the user role is "regular user," the semantic understanding model prioritizes matching elements operable within the current user's permission range (i.e., visible and not disabled elements) in the DOM summary table, skipping elements that require administrator privileges (such as buttons labeled "admin-only" or controls visible only to administrators). After completing the matching, the semantic understanding model arranges the operation steps according to the aforementioned execution order determination rules, generating a new sequence of operation instructions. Each operation step in this new sequence is adapted to the current execution context (for example, if the current execution context indicates that the user role is "regular user," the generated new sequence will not include operation steps requiring administrator privileges; if the current execution context indicates that the current date is the end of the month, the date-related input fields in the generated new sequence will be filled with the end-of-month date value). The operating system replaces the original sequence of operation instructions in the fixed operation instruction set with this new sequence and continues to execute subsequent operations using the new sequence.

[0084] During the execution of an operation instruction sequence (including the initial execution and the execution of a fixed operation instruction set), the operating system synchronously acquires the execution context information corresponding to the current operation step and establishes a correspondence between the operation instruction information of that operation step and the current execution context information. Specifically, when executing each operation step, the operating system synchronously collects the execution context information (including but not limited to the current webpage URL, current user identity, current time, etc.) at the time of execution of that operation step, associates this execution context information with the operation instruction information of that operation step (including operation type, target element location information, operation parameters or source path, etc.), forms a mapping record, and temporarily stores this mapping record in the operating system's memory. When the entire operation instruction sequence has been executed (i.e., all operation steps have been completed or terminated due to failure), the operating system arranges all the mapping records temporarily stored in memory during this execution in the execution order to form a complete mapping table, and persistently stores this mapping table in the fixed operation instruction set. Thus, when the fixed operation instruction set is subsequently called for context adaptation, the operating system can query the complete mapping table stored in the fixed operation instruction set.

[0085] In the context adaptation process, before invoking the semantic understanding model for re-parsing, the operating system first queries the mapping table stored in the fixed operation instruction set based on the current execution context information to determine if a mapping record matching the current execution context information exists. If a matching mapping record exists, the operating system directly updates the fixed operation instruction set based on the corresponding operation instruction information, replacing the original operation instruction information in the fixed operation instruction set with the operation instruction information from the mapping record, and then continues execution using the updated operation instruction information without invoking the semantic understanding model for re-parsing. If no matching mapping record exists in the mapping table, the operating system performs a re-matching step: it calls the semantic understanding model to re-parse the structured document data of the current webpage, generates a new sequence of operation instructions, and adds the correspondence between the current execution context information and the newly generated operation instruction information from this re-matching step as a new mapping record to the mapping table of the fixed operation instruction set, enriching the mapping table for direct querying in similar subsequent scenarios.

[0086] The specific execution process of the re-matching step is as follows: The operating system compares the current execution context information with the reference execution context information item by item, identifies the parameter items that differ between the two, and combines these differing parameter items and their difference values ​​into conditional deviation information. The conditional deviation information is used to characterize how the current execution environment has changed relative to the reference execution environment when it was fixed, such as "the user role has changed from an administrator to a regular user", "the current time has changed from a weekday to a holiday", "the current webpage URL has switched from the test environment to the production environment", etc.

[0087] The operating system obtains operation instruction information associated with the condition deviation information within the fixed operation instruction set. The operation instruction information includes at least one of the following: the source path of the dynamic data, the location information of the operation target, and the operation type. The value of the dynamic data changes with the execution context information. That is, the same dynamic data may need to be obtained from different data sources in different execution contexts (for example, when the user role is an administrator, the dynamic data is obtained from the administrator data source; when the user role is a regular user, the dynamic data is obtained from the regular user data source). The operating system determines the operation instruction information that needs to be adjusted based on the condition deviation information: for example, if the condition deviation information indicates "user role has changed," then the operation instruction information associated with this condition deviation information is the location information of the operation target (because the interface elements operable by users with different roles may be different); if the condition deviation information indicates "data source has changed," then the operation instruction information associated with this condition deviation information is the source path of the dynamic data (because the same dynamic data may need to be obtained from different source paths in different execution contexts).

[0088] The operating system inputs the original natural language operation intent, conditional deviation information, and the structured document data of the current webpage into the semantic understanding model. Specifically, the operating system obtains the complete DOM tree of the current webpage and generates a DOM summary table for the current webpage. It combines the original natural language operation intent text, the text description of the conditional deviation information (such as "The user role has changed from administrator to ordinary user, and the operation interface needs to be adapted for ordinary users"), and the DOM summary table of the current webpage into a prompt word and sends it to the semantic understanding model. Based on the original natural language operation intent and conditional deviation information, the semantic understanding model re-matches the operation instruction information corresponding to the conditional deviation information in the DOM summary table of the current webpage. For example, if the conditional deviation information indicates "The user role has changed from administrator to ordinary user," the semantic understanding model prioritizes matching elements operable within the current user's permission range in the DOM summary table of the current webpage, generating new operation instruction information adapted to the ordinary user role. If the conditional deviation information indicates "The dynamic data source path has changed," the semantic understanding model re-matches dynamic data source path elements semantically matching the original natural language operation intent in the DOM summary table of the current webpage, generating a new source path.

[0089] After the semantic understanding model completes the rematch, it generates new operation instruction information. The operating system replaces the original operation instruction information in the fixed operation instruction set with this new information. If the original operation instruction information is the location information of the operation target, it replaces the original location information with the newly matched information; if the original operation instruction information is the source path of dynamic data, it replaces the original source path with the newly matched path. After the operating system completes the replacement, it continues to execute subsequent operation steps using the updated fixed operation instruction set. The operating system adds the correspondence between the current execution context information and the newly generated operation instruction information in this rematch as a new mapping record to the mapping table of the fixed operation instruction set.

[0090] Taking the user-defined "Submit Project Application Form" operation as an example, the reference execution context information of this defined operation instruction set records the user role as "Administrator". During a certain call, the current user role in the current execution context information is "Regular User". The operating system compares the current execution context information with the reference execution context information, detects a difference in the "User Role" parameter, determines that the two do not match, and enters the context adaptation process. The operating system queries the mapping table of this defined operation instruction set. If there is no mapping record for the "Regular User" role in the mapping table, it enters the re-matching step. The operating system inputs the conditional deviation information "User role changed from Administrator to Regular User", the original natural language operation intent "Submit Project Application Form", and the DOM summary table of the current webpage into the semantic understanding model. Based on this conditional deviation information, the semantic understanding model re-matches the "Submit" button and other form elements operable within the regular user's permission range in the DOM summary table of the current webpage, generating a new sequence of operation instructions (for example, the "Approve" button, which was previously visible under the Administrator role, is not visible under the Regular User role, and the new sequence of operation instructions generated by the semantic understanding model no longer includes operation steps related to the "Approve" button). The operating system replaces the original operation instruction sequence in the fixed operation instruction set with a new sequence and adds the mapping relationship between the current execution context information (user role = ordinary user) and the newly generated operation instruction information adapted for ordinary users to the mapping table. When an ordinary user subsequently invokes this fixed operation instruction set, the operating system can directly query the corresponding mapping record stored in the mapping table and execute the operation instruction information adapted for the ordinary user directly, without needing to call the semantic understanding model again for re-parsing.

[0091] After the operating system completes the context adaptation process (whether it is a successful match directly through the mapping table or a new operation instruction information generated through a rematching step and then continues execution), it uses the adapted and fixed operation instruction set to execute subsequent operation steps until all operation steps are completed or a certain operation step fails and enters the fault handling process.

[0092] This application also discloses a natural language-driven web operating system. (See also...) Figure 2 ,include: Natural language receiving module 201 is used to receive natural language operation instructions input by the user; The instruction parsing and execution module 202 is used to respond to the natural language operation instruction by calling the semantic understanding model to parse the structured document data of the target webpage, generate an operation instruction sequence to operate on the target webpage, and execute the operation instruction sequence. The operation recording module 203 is used to record the operation information corresponding to each operation step in the operation instruction sequence during the execution of the operation instruction sequence; The operation solidification module 204 is used to respond to the solidification command triggered by the user, encapsulate the recorded operation information into a solidified operation command set, and associate and store the solidified operation command set with the URL information of the target webpage; The operation invocation module 205 is used to respond to the user's invocation command for the stored solidified operation instruction set, match the corresponding solidified operation instruction set according to the invocation command, and execute the operation instruction sequence recorded in the solidified operation instruction set.

[0093] This application also discloses a natural language-driven webpage operation device, which includes a memory and a processor. The memory stores a computer program that can be loaded by the processor and executed as described above for the natural language-driven webpage operation method.

[0094] This application also discloses a computer-readable storage medium that stores a computer program that can be loaded by a processor and executed as described above in the natural language-driven web page operation method. The computer-readable storage medium includes, for example, various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0095] It should be noted that in this paper, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0096] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit the scope of protection of the application. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on these embodiments, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.< / form> < / button>

Claims

1. A natural language-driven webpage operation method, characterized in that, include: Receive natural language operation instructions input by the user; In response to the natural language operation instructions, a semantic understanding model is invoked to parse the structured document data of the target webpage, generate a sequence of operation instructions to operate on the target webpage, and execute the sequence of operation instructions; During the execution of the operation instruction sequence, the operation information corresponding to each operation step in the operation instruction sequence is recorded; In response to a user-triggered fixed command, the recorded operation information is encapsulated into a fixed operation command set, and the fixed operation command set is associated with and stored with the URL information of the target webpage; In response to a user's call command for a stored set of solidified operation instructions, the corresponding set of solidified operation instructions is matched according to the call command, and the sequence of operation instructions recorded in the set of solidified operation instructions is executed.

2. The natural language-driven webpage operation method according to claim 1, characterized in that, The method further includes: When executing the sequence of operation instructions recorded in the solidified operation instruction set, if the operation information recorded in the solidified operation instruction set cannot locate the corresponding operation target in the structured document data of the current webpage, then the original natural language operation intent corresponding to the solidified operation instruction set is extracted. Based on the original natural language operation intent, the semantic understanding model is invoked to re-parse the structured document data of the current webpage, generate a new sequence of operation instructions, and replace the operation information recorded in the solidified operation instruction set with the new sequence of operation instructions.

3. The natural language-driven webpage operation method according to claim 1, characterized in that, The method further includes: During the execution of the operation instruction sequence, dynamic data in the operation instruction sequence is identified, and the source path of the dynamic data is obtained, wherein the source path is used to indicate the way to obtain the current value of the dynamic data when the fixed operation instruction set is executed; In response to a user-triggered fixed instruction, the source path and the operation steps corresponding to the dynamic data are associated and stored in the fixed operation instruction set; When executing the solidified operation instruction set, the current value of the dynamic data is obtained in real time according to the source path, and the corresponding operation steps are executed.

4. The natural language-driven webpage operation method according to claim 3, characterized in that, The source path for obtaining the dynamic data includes: When the dynamic data corresponds to multiple source paths, the original natural language operation intent corresponding to the fixed operation instruction set is obtained, and the semantic matching degree between the original natural language operation intent and the data semantic tag associated with each source path is calculated. The source path with the highest semantic matching degree is taken as the source path of the dynamic data.

5. The natural language-driven webpage operation method according to claim 3, characterized in that, The fixed operation instruction set is also associated with and stored reference execution context information when the operation instruction sequence is executed; The execution of the sequence of operation instructions recorded in the solidification operation instruction set includes: Obtain the current execution context information and the reference execution context information corresponding to the fixed operation instruction set; verify whether the current execution context information matches the reference execution context information; If a match is found, the sequence of operation instructions recorded in the solidified operation instruction set is executed; If there is no match, the original natural language operation intent corresponding to the solidified operation instruction set is obtained. Based on the original natural language operation intent and the current execution context information, the structured document data of the current webpage is re-parsed to generate a new operation instruction sequence adapted to the current execution context information. The corresponding operation instruction sequence in the solidified operation instruction set is replaced with the new operation instruction sequence and then executed.

6. The natural language-driven webpage operation method according to claim 5, characterized in that, The step of re-parses the structured document data of the current webpage based on the original natural language operation intent and the current execution context information to generate a new sequence of operation instructions adapted to the current execution context information, including a re-matching step: The current execution context information is compared with the reference execution context information to extract condition deviation information; Obtain operation instruction information associated with the condition deviation information from the solidified operation instruction set. The operation instruction information includes at least one of the following: the source path of dynamic data, the location information of the operation target, and the operation type; wherein, the value of the dynamic data also changes with the execution context information. The original natural language operation intent, the conditional bias information, and the structured document data of the current webpage are input into the semantic understanding model. The semantic understanding model then uses the original natural language operation intent and the conditional bias information to re-match new operation instruction information corresponding to the operation instruction information in the structured document data of the current webpage. The original operation instruction information in the solidified operation instruction set is replaced with the new operation instruction information to update the solidified operation instruction set.

7. The natural language-driven webpage operation method according to claim 6, characterized in that, The method further includes: During the execution of the operation instruction sequence, the execution context information corresponding to the operation instruction information is obtained, and a mapping relationship is established between different operation instruction information corresponding to different execution context information; the mapping relationship is stored in the fixed operation instruction set. The step of re-parses the structured document data of the current webpage based on the original natural language operation intent and the current execution context information to generate a new sequence of operation instructions adapted to the current execution context information further includes: The mapping relationship is queried first based on the current execution context information. If there is a mapping record in the mapping relationship that matches the current execution context information, the fixed operation instruction set is updated directly based on the operation instruction information corresponding to the mapping record. If there is no mapping record in the mapping relationship that matches the current execution context information, the re-matching step is performed, and the correspondence between the execution context information and the operation instruction information generated in this re-matching step is added to the mapping relationship as a new mapping record.

8. A natural language-driven web page operating system, characterized in that, include, The natural language receiving module (201) is used to receive natural language operation instructions input by the user; The instruction parsing and execution module (202) is used to respond to the natural language operation instruction by calling the semantic understanding model to parse the structured document data of the target webpage, generate an operation instruction sequence to operate on the target webpage, and execute the operation instruction sequence; The operation recording module (203) is used to record the operation information corresponding to each operation step in the operation instruction sequence during the execution of the operation instruction sequence; The operation solidification module (204) is used to respond to the solidification command triggered by the user, encapsulate the recorded operation information into a solidified operation command set, and store the solidified operation command set in association with the URL information of the target webpage; The operation invocation module (205) is used to respond to the user's invocation command for the stored solidified operation instruction set, match the corresponding solidified operation instruction set according to the invocation command, and execute the operation instruction sequence recorded in the solidified operation instruction set.

9. A natural language-driven webpage operation device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer program is stored that can be loaded by a processor and executed as described in any one of claims 1 to 7.

Images