Browser automation method and system based on natural language processing

Abstract

The invention relates to the technical field of natural language processing, in particular to a browser automation method and system based on natural language processing. The invention particularly relates to a method and a system for generating an operation instruction by simplifying a webpage DOM (Document Object Model) structure and utilizing a GPT model. By monitoring and adapting to webpage structure changes in real time, the system can keep continuous effectiveness of automatic operation. Even if the page is subjected to large structure adjustment, the system can still automatically analyze and adjust the operation logic, and the intervention requirement of a user is reduced. And the circular updating and adapting module can quickly respond to webpage changes, so that the phenomenon of operation failure or interruption caused by page updating is reduced. Even if the DOM structure of the page is greatly changed, the system can quickly adjust, and successful execution of the task is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a browser automation method and system based on natural language processing; specifically, a method and system for generating operation instructions by simplifying the DOM structure of a web page and utilizing a GPT model. Background Art

[0002] In the Internet age, users frequently perform various repetitive tasks in browsers, such as filling out forms, submitting data, clicking links, and navigating pages. These operations are often cumbersome and time-consuming, especially when dealing with complex web applications, where users need to repeat similar steps over and over again. Although traditional browser automation tools can partially solve these problems, they usually require writing fixed operation scripts or macro commands. This approach has the following main problems: 1. Lack of flexibility: Fixed scripts cannot handle dynamically changing web page content and complex interactive tasks, and are difficult to adapt to updates and changes in web page content.

[0003] 2. Privacy risks: Some tools need to upload data to remote servers, which may lead to privacy leakage and security risks of user data.

[0004] 3. Operational complexity: Writing and maintaining detailed operation scripts increases operational complexity and maintenance costs.

[0005] In recent years, advances in natural language processing technology have provided new solutions. In particular, technologies based on advanced language models such as GPT-4 can understand and generate natural language text, making browser automation more intelligent and convenient. GPT-4 can parse users' natural language instructions and generate corresponding operation steps based on these instructions, thus significantly improving the intelligence and flexibility of automation tools. Summary of the invention

[0006] The main purpose of the present invention is to provide a browser automation method and device based on natural language processing to achieve the following goals: 1. Intelligent command analysis: Use natural language processing technology to intelligently analyze users' natural language commands, and automatically generate and execute browser operations.

[0007] 2. Automatic processing of complex tasks: Supports automatic processing of various complex web page operations, including form filling, button clicking, page navigation, etc., to adapt to dynamically changing web page content.

[0008] 3. Protect user privacy: All operations are completed locally without involving remote servers, ensuring the privacy and security of user data.

[0009] 4. Improve work efficiency: By automating repetitive tasks, the time and energy invested in manual operations can be reduced, and the overall work efficiency and task accuracy can be improved.

[0010] Based on natural language processing, the browser automation method is as follows: the user inputs natural language instructions and describes the operations that the user wants to perform in natural language; natural language processing parses the instructions, identifies the intent, and generates operation steps; the web page DOM is parsed to extract the web page structure and interactive elements, providing the basis for operation; the web page elements are interacted with according to the generated operation steps; the operation status is monitored in real time, exceptions are handled, and strategies are adjusted; the model is updated according to changes in the web page; Web page DOM parsing: Get the complete HTML structure of a web page, parse and simplify it, extract elements related to user interaction, such as buttons, input boxes, links; parse the HTML document of a web page, including static content and dynamically loaded content; a. Parsing of static content Get the HTML document of the web page directly through HTTP request. This is the basic way to parse static content. Use tools such as curl, requests, and BeautifulSoup to grab and parse HTML content; use the parser BeautifulSoup or lxml to convert the obtained HTML document into a DOM tree and extract all elements in it, including nested structures; in the DOM tree, extract the attributes of each node, such as id, class, name, href, src, to provide positioning information for subsequent operations; b. Parsing of dynamic content Modern web pages often contain dynamic content loaded via JavaScript or AJAX; use browser automation tools such as Selenium or Puppeteer to simulate user behavior and set wait conditions WebDriverWait or waitForSelector to wait for dynamic elements in the page to load; directly execute the JavaScript code of the page, or use interfaces such as fetch and XMLHttpRequest in the window object to obtain dynamically loaded data; in single-page applications, page content is loaded asynchronously through routing and components; identify changes in page content by monitoring pushState, popState, or hashchange events of the browser's history object. Use MutationObserver to monitor changes in the DOM to ensure that elements dynamically inserted through JavaScript are captured.

[0011] The web page DOM parsing simultaneously performs web page DOM simplification, uses natural language processing (NLP) technology to perform semantic analysis on the text content in the web page, identifies the content related to the user instruction and forms a DOM tree, and the DOM tree is simplified to prioritize the retention of content.

[0012] The system of the browser automation method based on natural language processing includes a natural language instruction parsing module, a web page DOM parsing module, an operation execution module, a state monitoring and feedback module and a loop update and adaptation module; the natural language instruction parsing module uses the natural language processing technology GPT-4 to parse the natural language instructions input by the user; the module can recognize and understand the user's intention and convert the natural language instructions into specific operation steps; the module includes functions such as instruction parsing, intention recognition, and command generation; the web page DOM parsing module obtains and parses the HTML structure of the web page and converts it into a simplified DOM tree; the module extracts elements related to user interaction, such as buttons, input boxes, and links to provide basic data for operation execution; the operation execution module performs interactive operations on web page elements according to the parsed operation steps, including clicking buttons, setting input box values, submitting forms, etc., and the module is realized by directly interacting with DOM elements; the state monitoring and feedback module monitors the page status in real time and evaluates the completion of the operation; the operation strategy is dynamically adjusted according to the feedback information to deal with possible abnormal situations or errors. The loop update and adaptation module updates the natural language processing model according to the new DOM structure and operation history. The system can dynamically adapt to changes in web page content and structure and continue to perform subsequent operations.

[0013] The beneficial effects of the present invention are: (1) Continuous and effective automated operations: By real-time monitoring and adapting to changes in web page structure, the system can maintain the continuous effectiveness of automated operations. Even if the page undergoes major structural adjustments, the system can still automatically parse and adjust the operation logic, reducing the need for user intervention.

[0014] (2) Reduce operation failures and interruptions: The loop update and adaptation module can quickly respond to changes in web pages and reduce operation failures or interruptions caused by page updates. Even if the DOM structure of the page changes significantly, the system can quickly adjust to ensure the successful execution of the task.

[0015] (3) Improve long-term stability: The system gradually optimizes the operation logic and adapts to the changes of different web pages through operation history recording and learning. This makes the system more stable and adaptable in the long-term use process, reducing the user's need for script maintenance.

[0016] (4) Reduce maintenance costs: Since the system can automatically adapt to changes in web pages, users do not need to frequently maintain or update automated scripts, which reduces maintenance costs. Especially when web pages are frequently updated, this module can significantly reduce the need for manual intervention.

[0017] (5) Reduce the need for frequent script updates: In traditional automation tools, whenever a web page changes, users need to manually modify the script to adjust element positioning and operation logic. Through the cyclic update and adaptation module, the system can automatically adapt to page changes, greatly reducing the need for users to manually maintain scripts.

[0018] (6) Reduce the workload of technical personnel: Traditional scripting tools often require users with technical backgrounds to write and maintain scripts. As web pages are constantly updated, technical personnel need to frequently adjust and test them. The present invention reduces the workload of technical personnel by automatically adapting to web page changes, allowing non-technical personnel to easily use automated tools. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of the relationship between components of the system; Figure 2 This is a natural language processing flow chart. DETAILED DESCRIPTION

[0020] Based on natural language processing, the browser automation method is as follows: the user inputs natural language instructions and describes the operations that the user wants to perform in natural language; natural language processing parses the instructions, identifies the intent, and generates operation steps; the web page DOM is parsed to extract the web page structure and interactive elements, providing the basis for operation; the web page elements are interacted with according to the generated operation steps; the operation status is monitored in real time, exceptions are handled, and strategies are adjusted; the model is updated according to changes in the web page; Web page DOM parsing: Get the complete HTML structure of a web page, parse and simplify it, extract elements related to user interaction, such as buttons, input boxes, links; parse the HTML document of a web page, including static content and dynamically loaded content; a. Parsing of static content Get the HTML document of the web page directly through HTTP request. This is the basic way to parse static content. Use tools such as curl, requests, and BeautifulSoup to grab and parse HTML content; use the parser BeautifulSoup or lxml to convert the obtained HTML document into a DOM tree and extract all elements in it, including nested structures; in the DOM tree, extract the attributes of each node, such as id, class, name, href, src, to provide positioning information for subsequent operations; b. Parsing of dynamic content Modern web pages often contain dynamic content loaded via JavaScript or AJAX; use browser automation tools such as Selenium or Puppeteer to simulate user behavior and set wait conditions WebDriverWait or waitForSelector to wait for dynamic elements in the page to load; directly execute the JavaScript code of the page, or use interfaces such as fetch and XMLHttpRequest in the window object to obtain dynamically loaded data; in single-page applications, page content is loaded asynchronously through routing and components; identify changes in page content by monitoring pushState, popState, or hashchange events of the browser's history object. Use MutationObserver to monitor changes in the DOM to ensure that elements dynamically inserted through JavaScript are captured.

[0021] The web page DOM parsing simultaneously performs web page DOM simplification, uses natural language processing (NLP) technology to perform semantic analysis on the text content in the web page, identifies the content related to the user instruction and forms a DOM tree, and the DOM tree is simplified to prioritize the retention of content.

[0022] The system of the browser automation method based on natural language processing includes a natural language instruction parsing module, a web page DOM parsing module, an operation execution module, a state monitoring and feedback module and a loop update and adaptation module; the natural language instruction parsing module uses the natural language processing technology GPT-4 to parse the natural language instructions input by the user; the module can recognize and understand the user's intention and convert the natural language instructions into specific operation steps; the module includes functions such as instruction parsing, intention recognition, and command generation; the web page DOM parsing module obtains and parses the HTML structure of the web page and converts it into a simplified DOM tree; the module extracts elements related to user interaction, such as buttons, input boxes, and links to provide basic data for operation execution; the operation execution module performs interactive operations on web page elements according to the parsed operation steps, including clicking buttons, setting input box values, submitting forms, etc., and the module is realized by directly interacting with DOM elements; the state monitoring and feedback module monitors the page status in real time and evaluates the completion of the operation; the operation strategy is dynamically adjusted according to the feedback information to deal with possible abnormal situations or errors. The loop update and adaptation module updates the natural language processing model according to the new DOM structure and operation history. The system can dynamically adapt to changes in web page content and structure and continue to perform subsequent operations.

[0023] The specific technical solution includes the following steps and components: 1. Web page DOM analysis: Step description: Get the complete HTML structure of the web page, parse and simplify it, and extract elements related to user interaction, such as buttons, input boxes, links, etc.

[0024] (1) Specific requirements for web page DOM parsing: DOM parsing is the basis of web page automation. Parsing needs to cover the following requirements: Must be able to fully parse the HTML document of the web page, including static content and dynamically loaded content (such as elements loaded via JavaScript or AJAX). Ensure that the parsing supports the DOM structure of all major browsers, including Chrome, Firefox, Safari, etc.

[0025] The specific requirements for HTML structure parsing are described as follows: a. Parsing of static content Directly obtain the source code of the web page: directly obtain the HTML document of the web page through HTTP request. This is the basic way to parse static content. You can use tools such as curl, requests, and BeautifulSoup to crawl and parse HTML content. DOM tree construction: use a parser (such as BeautifulSoup, lxml) to convert the obtained HTML document into a DOM tree and extract all elements in it, including nested structures. Element attribute extraction: in the DOM tree, extract the attributes of each node (such as id, class, name, href, src, etc.) to provide positioning information for subsequent operations.

[0026] b. Parsing of dynamic content Handling JavaScript and AJAX loading: Modern web pages often contain dynamic content loaded via JavaScript or AJAX. This content will not be displayed until the page is loaded or an event is triggered. Waiting mechanism: Use browser automation tools (such as Selenium, Puppeteer) to simulate user behavior and set waiting conditions (such as WebDriverWait or waitForSelector) to wait for dynamic elements in the page to load. Execute scripts to obtain dynamic content: Directly execute the JavaScript code of the page, or use interfaces such as fetch, XMLHttpRequest in the window object to obtain dynamically loaded data. Handling SPA (single-page application): In a single-page application, page content is loaded asynchronously through routes and components. Listening for route changes: Identify changes in page content by monitoring the browser's history object (pushState, popState) or hashchange events. DOM Mutation Observers: Use MutationObserver to monitor changes in the DOM to ensure that elements dynamically inserted through JavaScript are captured.

[0027] Specific definition and subsequent analysis and simplification a. Defining completeness Page loading monitoring: The standard for page loading is that the DOM content has been fully rendered, including the dynamically loaded parts. It can be defined in the following ways: DOMContentLoaded event: indicates that the initial static content is loaded, but does not include the subsequent dynamically loaded parts. Load event: indicates that the entire page, including static resources (such as images, style sheets), has been loaded. Custom conditions: For complex pages, monitor the insertion of dynamic content through MutationObserver, combined with AJAX request monitoring (such as listening to XMLHttpRequest, fetch), to ensure that all data is loaded.

[0028] b. Subsequent parsing and simplification Simplify the DOM tree: Filter irrelevant elements: According to preset rules, filter out elements that do not participate in user interaction (such as advertisements, decorative images, invisible elements). Extract key elements: Keep elements that users may interact with (such as <input> , <button>、 、 <form>wait).

[0029] Dynamic adjustment: Retry mechanism: When the page content is dynamically updated, re-execute DOM parsing to ensure that the parsed structure remains up to date. Cache and update: Cache the parsed DOM structure and trigger the re-parsing process when the page structure changes significantly.

[0030] Through the above technical requirements and parsing process, the system can completely obtain and parse the HTML document of the web page, including static and dynamic content, and provide accurate structural data for subsequent automated operations.

[0031] Dynamic content processing: For dynamic content loaded by JavaScript in modern web pages, the parsing system should have the ability to delay loading. You can monitor network requests, page events, or set a timeout to ensure that the page is loaded before parsing. You can consider using tools (such as Puppeteer, Selenium, etc.) to handle SPA (single-page applications) or elements loaded asynchronously via AJAX.

[0032] Deep parsing of the DOM tree: The parsing process needs to support deep traversal of the DOM tree, parsing each child element layer by layer to ensure that no nested interactive elements (such as nested forms, buttons, links, etc.) are missed. The parsing of elements must include their attributes (such as id, class, name, href, src, etc.) so that subsequent automated operations can locate and interact.

[0033] Compatibility requirements: Support parsing of multiple webpage languages ​​(such as HTML5, XHTML, XML). It should be compatible with webpage structures generated by different CSS and JS frameworks (such as React, Vue, Angular), and ensure that the elements rendered by the framework can be correctly parsed.

[0034] The process of simplifying the DOM is an important step to improve system efficiency and accuracy. The following are specific technical standards for simplifying operations: Key element extraction criteria: only retain DOM elements that are closely related to user interaction, such as input boxes ( <input> ), button(< / form> < / button> <button> ), Link( ), and forms(), ignoring decorative elements not relevant to automation (such as , , <header>When extracting these key elements, you should rely on the characteristic attributes of the elements (such as id, class, name, etc.) to ensure the uniqueness of the elements and the accuracy of the operation. Redundant element removal standard: The simplification process needs to remove redundant, repeated or irrelevant elements (such as advertisements, decorative images, non-functional DIVs, etc.) to reduce the complexity of the DOM tree. In addition, elements that are hidden or invisible according to CSS rules (such as display: none;) are automatically ignored during simplification.

[0035] DOM structure optimization standards: In the simplified DOM tree, the hierarchical relationship between all interactive elements should be clear. Reorganize the original DOM tree to remove unnecessary nested relationships to ensure a simpler operation process. Use optimization algorithms to reduce the depth and width of the DOM tree to reduce the complexity of traversing the DOM tree during subsequent operations. For example, move deeply nested form elements to a shallower level to find the target element faster.

[0036] Dynamically adapt to web page changes: When the web page content is updated or reloaded, the system should be able to dynamically adjust the DOM structure to adapt to the changes in the web page. To this end, a cache mechanism can be added during the simplification process, and when the DOM structure changes significantly, it will be re-parsed and simplified.

[0037] Example 1: Enterprise Internal Automated Office Scenario description: In an enterprise, employees need to frequently submit weekly reports in the company's internal system. Through the device of the present invention, the whole process of logging in, navigating to the weekly report submission page, filling in the content and submitting can be automatically completed.

[0038] Beneficial effects: Automated processing significantly improves the efficiency of weekly report submission, reduces the time and errors of manual operations, and improves employee work efficiency and the accuracy of report submission.

[0039] Example 2: Automated management of online education platform Scenario description: On an online education platform, teachers need to generate personalized learning reports for each student. The device of the present invention can automatically complete the whole process of logging in, navigating to the student management page, generating reports and sending them.

[0040] Beneficial effects: Teachers can quickly generate and send learning reports, reducing the workload of manual operations, improving teaching efficiency and the accuracy of report generation. Teachers can devote more time to teaching activities and optimize teaching results.

[0041] Example 3: Automated management of e-commerce platforms Scenario description: On an e-commerce platform, administrators need to regularly check product inventory and generate inventory reports. The device of the present invention can automatically complete the entire process of logging in, navigating to the inventory management page, checking inventory, and generating reports.

[0042] Beneficial effects: Administrators can efficiently manage inventory and ensure the accuracy and timeliness of inventory data. Automated management reduces the time and errors of manual inventory checks and improves the efficiency and accuracy of inventory management. Preferred Embodiments

[0043] Preferred embodiment: Automated processing of enterprise customer relationship management (CRM) system Scenario description: In an enterprise CRM system, sales personnel need to regularly update customer information, generate sales reports, and schedule meetings. Using the device of the present invention, it is possible to automatically log into the CRM system, navigate to the customer information page, update customer data, generate sales reports, and schedule meetings.

[0044] Beneficial effects: Automating these tasks reduces sales staff’s time on repetitive data entry and report generation, allowing them to focus more on customer interaction and sales strategy formulation, significantly improving sales efficiency and customer satisfaction.

[0045] Expanding technical solutions Expanded technical solutions: Automation support in multiple browser environments Scenario description: In a complex working environment, users may need to operate multiple browsers or multiple tabs at the same time. The extended technical solution includes supporting the automated processing of multiple browsers and multiple tabs.

[0046] Technical implementation: Design a cross-browser and cross-tab management module that can monitor and coordinate multiple browser instances to ensure that tasks in different browsers and tabs can be seamlessly connected and executed synchronously. Use browser identifiers and session management technology to maintain the status information of each browser instance.

[0047] Beneficial effects: The adaptability and practicability of the present invention are enhanced, enabling it to provide efficient automated services in more complex working environments, and improving the work efficiency and task coordination ability of users in multi-tasking.

[0048] Technical points: The core technical solution of the present invention is developed around a browser automation method and device based on natural language processing. The specific technical points are as follows: 1. Natural language instruction parsing module Technical highlights: Use advanced natural language processing technology (such as GPT-4) to parse the natural language instructions input by users. This module can recognize and understand the user's intentions and convert natural language instructions into specific operation steps. The module includes functions such as instruction parsing, intention recognition, and command generation.

[0049] Beneficial effects: It reduces the user's operating threshold, enables users to describe complex operations in simple natural language, and avoids the tedious script writing process. It improves the flexibility and intelligence level of operation.

[0050] 2. Web page DOM parsing module Technical points: Get and parse the HTML structure of the web page and convert it into a simplified DOM tree. This module extracts elements related to user interaction (such as buttons, input boxes, links, etc.) to provide basic data for operation execution.

[0051] Beneficial effects: Simplifies the processing of complex web pages, improves the accuracy and efficiency of operations, and ensures that the system can effectively identify and process key interactive elements in web pages.

[0052] 3. Operation execution module Technical points: According to the parsed operation steps, interactive operations are performed on web page elements, including clicking buttons, setting input box values, submitting forms, etc. The module is implemented by directly interacting with DOM elements.

[0053] Beneficial effects: It provides an intuitive and easy-to-use operation method, making the automation process more efficient and reliable. It reduces the time and error rate of manual operation and improves the accuracy of operation.

[0054] 4. Condition monitoring and feedback module Technical points: Monitor page status in real time and evaluate the completion of operations. Dynamically adjust operation strategies based on feedback information to deal with possible abnormal situations or errors.

[0055] Beneficial effects: Ensure the stability of the operation process and the accurate completion of tasks. Ability to handle abnormal situations and errors in real time, ensuring the reliability and consistency of automated operations.

[0056] 5. Cyclic update and adaptation module Technical points: Update the natural language processing model according to the new DOM structure and operation history. The system can dynamically adapt to changes in web page content and structure and continue to perform subsequent operations.

[0057] Beneficial effects: Ensure the consistency and effectiveness of the automation process, reduce operational interruptions and failures caused by web page updates, and improve the system's adaptability and long-term stability.

[0058] Specific technical effects: 1. Improve operational efficiency: By automating complex web page operations, the need for manual intervention is reduced, thereby significantly improving operational efficiency. Users can complete a large number of repetitive tasks in a shorter period of time, saving time and energy.

[0059] Specific requirements and supporting evidence for improving operational efficiency a) Complex web pages usually involve dynamically loaded content, nested DOM structures, multiple forms, buttons, links, and page jumps. There may be complex interaction logic and dependencies between these elements.

[0060] b) The automation system needs to have: Dynamic content processing capabilities: support the processing of dynamic content loaded through JavaScript or Ajax to ensure the consistency and accuracy of operations. Multi-step task execution: support the execution of multi-step tasks, such as submitting a form first and then navigating to other pages. Cross-page operations: be able to handle operations after page jumps, such as jumping to the dashboard or data management page after logging in. Status monitoring and feedback adjustment: The automation system should monitor the status of the web page in real time during execution, handle page loading delays or operation failures, and make corresponding adjustments.

[0061] (2) Specific examples to support the improvement of operational efficiency: Case 1: Automatic submission of weekly reports within the enterprise: In manual operation, employees need to log in to the system, fill out the form and submit it every week. This operation takes an average of 10-15 minutes. With this automated system, employees only need to initiate the operation through natural language instructions once, and the system automatically completes all steps. The whole process only takes 1-2 minutes, saving more than 80% of time. Evidence: Through time record comparison, it is proved that the automated system significantly shortens the operation time and avoids employees' repetitive manual input and page navigation.

[0062] Case 2: E-commerce platform inventory check and report generation: Administrators need to log in to the e-commerce backend to manually check the inventory of each product and generate reports. Usually, this task involves hundreds of products and takes hours to complete manually. The automated system can complete the inventory check and report generation of all products in a few minutes. Especially when the page content is dynamically loaded, the system automatically waits for and processes the page feedback to ensure the accuracy of the data. Evidence: The task execution time is shortened from 3 hours for manual operation to 10 minutes for the automated system. This significant efficiency improvement can be verified through automated logging.

[0063] Case 3: Generation and delivery of student reports on online education platforms: Teachers manually generate and send each student's learning report. This process includes logging into the platform, entering the student management page, generating reports, and sending them one by one, which is very time-consuming. The automated system can perform these operations in batches, reducing the task time of generating and sending reports from 1 hour to less than 5 minutes, ensuring that all reports are accurately generated and sent. Evidence: The efficiency of the automated system is proven by comparing the task execution log and the report generation speed.

[0064] 2. Reduce operational complexity: The natural language processing module enables users to describe complex operations in simple natural language without writing complex scripts or macro commands. This lowers the threshold for operation and reduces technical background requirements.

[0065] Specific instructions to reduce operational complexity By introducing the natural language processing (NLP) module, the present invention allows users to use natural language instructions to perform complex web page operations, eliminating the need to write complex scripts or macro commands, and lowering the threshold for use. The following will further explain how the natural language processing module achieves this function, and illustrate the effect of reducing operation complexity with specific cases.

[0066] Working principle and implementation of natural language processing module (1). Natural language instruction parsing: Users can input commands in natural language, such as "log in to the system and submit a report" or "check inventory and generate a report". The NLP module analyzes the key operation words in the command (such as "log in", "submit", "generate", etc.) and the target objects (such as "system", "report", "inventory"). The module uses GPT-4 or similar language models to parse user instructions, extract clear operation intentions, and convert them into operation steps that the system can understand.

[0067] (2) Intent Recognition: The NLP module can not only recognize explicit instructions, but also infer the user's operation intention from ambiguous or incomplete instructions. For example, if the user enters "help me submit today's weekly report", the system can automatically infer that the operation is to log in to the system, fill in the weekly report content and submit the form.

[0068] (3). Command generation: The NLP module converts the parsed user intent into clear operation commands and executes them through the browser automation module. The generated commands include specific operation steps such as click, fill, navigate, submit, etc., avoiding manual intervention by users.

[0069] Specific implementation steps: (1). User input: Users enter natural language instructions through text boxes, such as "Log in and fill in the weekly report" (2). Language model analysis: The GPT-4 model analyzes the instructions and breaks them down into specific operations. For example, "Log in" is parsed as "locate the username and password input boxes and enter the information, click the login button", and "fill in the weekly report" is parsed as "find the weekly report text box and fill in the content, click the submit button". (3). Operation step generation: The system automatically generates operation steps and executes them through the browser automation module. Users do not need to understand the complex structure behind the web page, nor do they need to write operation scripts, which greatly reduces the complexity of the operation.

[0070] Use cases to illustrate the effect of reducing operational complexity Case 1: Automatically log in and submit weekly reports Traditional operation complexity: Employees need to manually open the browser, log in to the system, find the weekly report page, fill in and submit the report. In order to achieve automation, users need to write complex scripts or use macro-like tools to configure page operations, which requires a certain technical background and scripting knowledge. NLP automation effect: Through the natural language processing module, users only need to enter a simple command "Help me submit the weekly report", and the system will automatically complete all operations such as logging in, filling in and submitting. Users do not need to understand the background DOM structure, form attributes or write any scripts.

[0071] Case 2: Generate and send student reports in batches Traditional operation complexity: Teachers need to log in to the online education platform, generate reports for each student one by one and send them manually. This involves multi-step page operations. If batch operations are implemented through traditional automation tools, complex scripts need to be configured. NLP automation effect: Teachers use natural language to input "generate and send learning reports for all students", and the system automatically parses the instructions and executes the steps of generation and sending step by step. The system can automatically handle cross-page operations and multiple tasks, and users do not need to deal with complex technical details.

[0072] Case 3: E-commerce platform inventory check and report generation Traditional operation complexity: E-commerce administrators need to log in to the backend, manually check the inventory data of each product and generate reports. To automate operations with traditional automation tools, users need to pre-write scripts, specify page elements, and configure loop logic. NLP automation effect: Administrators only need to enter "check all product inventory and generate reports", and the system automatically parses out operations such as login, navigation, checking inventory, and generating reports. Users do not need to understand complex web page operation logic, nor do they need to write any scripts.

[0073] The beneficial effect of reducing operational complexity (1) No technical background required: Users can use natural language to input operation instructions without writing code or scripts. This greatly reduces the technical threshold for operation, allowing users without programming skills to use automation tools.

[0074] (2) Simplify complex tasks: The system can break down complex multi-step operations into simple natural language commands, allowing users to easily complete tasks that originally required scripting. For example, complex cross-page operation tasks can be completed through natural language instructions, avoiding tedious manual operations.

[0075] (3) Improve user experience: Using natural language as an input method conforms to the user's daily expression habits, reduces the complexity of operation, and improves the user experience. Users can describe complex tasks through simple instructions, and the system will automatically parse and execute them.

[0076] 3. Protect user privacy: All operations are completed locally, without uploading user data to a remote server. This design ensures user privacy and avoids the risk of data leakage and abuse.

[0077] 4. Improve operation accuracy: Real-time monitoring and feedback mechanisms can handle abnormal situations that may occur during operation and ensure accurate completion of tasks. This improves the reliability and stability of automated operations.

[0078] 5. Adapt to dynamic changes: The cyclic update and adaptation module can handle changes in web page content and structure, ensuring the continued effectiveness of the automation process. It reduces operational failures and interruptions caused by web page updates and improves the long-term stability of the system.

[0079] The "loop update and adaptation module" is an important part of the present invention, which is designed to handle changes in web page content and structure, ensure the continued effectiveness of automated operations, and avoid operation failures or interruptions caused by web page updates or content changes. This module automatically detects changes in the web page DOM structure, updates the operation logic, and combines the operation history to ensure that the system can operate stably in the long term.

[0080] Working principle and implementation of the cyclic update and adaptation module (1). Dynamically monitor web page changes: The cyclic update and adaptation module monitors the DOM structure changes of the web page in real time, especially those dynamically generated content (such as those loaded by JavaScript or Ajax). It can identify new elements added after the page is loaded, as well as changes in the attributes of certain elements (such as changes in the position of buttons and input boxes).

[0081] (2). Automatic update operation logic: When the DOM structure of a web page changes (such as updated web page tags, form field position changes), the module can automatically adapt to the new structure and update the operation instructions. For example, a new button may have a changed position or ID, but the module can automatically locate the new button and update the operation logic by re-parsing the page.

[0082] (3). Operational history and learning: The system will record the historical data of each operation and analyze the execution of the operation. Through the operation history, the system can gradually learn the user's common operation mode and give priority to trying historical operation steps when the web page structure changes, reducing the time and resource consumption of re-parsing.

[0083] (4). Cyclic update mechanism: After each operation, the system will re-evaluate and update the DOM structure to ensure the consistency and effectiveness of the operation. Especially in complex multi-page tasks, this module ensures the consistency and accuracy of the operation logic in different pages.

[0084] Specific implementation steps of the cyclic update and adaptation module (1). Web page change monitoring: The system continuously monitors the DOM structure of the page and uses change detection algorithms (such as node tree changes, element attribute changes, etc.) to determine whether the page content has changed. For dynamically generated content (such as loaded via Ajax), the system can wait until the page is loaded before operating.

[0085] (2). Re-parse and update the operation logic: If the system detects that the web page has changed, it will reparse the new DOM structure and update the relevant operation instructions. For example, after the page is upgraded, the position or ID of the button changes, and the system can automatically reposition and execute the click operation.

[0086] (3) Historical operation optimization: The system optimizes its adaptation to the new structure based on the data in the operation history. For example, if the same operation has been performed in a similar page structure, the system will give priority to trying to use the successful operation logic to reduce parsing time and improve operation stability.

[0087] (4) Feedback and Adjustment: The loop update module collaborates with the status monitoring module. When an operation fails, the system will dynamically adjust the strategy based on the feedback information, re-identify or parse the page to ensure the smooth execution of the operation.

[0088] Using actual cases to illustrate the effect of adapting to dynamic changes (1) Case 1: Update of the login page of the enterprise internal system Problem description: An enterprise updated the login page of its internal system. The position and labels of the form fields changed, but employees still needed to log in and submit data through automated tools.

[0089] Problems with traditional automation tools: Old automation tools cannot recognize new form labels or buttons, resulting in interrupted operations and users need to manually modify the operation scripts.

[0090] Cyclic update module adaptability: The system monitors the changes in the DOM structure of the web page in real time and finds that the labels and positions of the username, password input boxes and login button have changed. The module automatically re-parses the page and updates the automation instructions to ensure that the new input boxes and buttons can be correctly located to complete the login operation. Effect: Employees do not need to modify any scripts, and the automation process can still adapt to the new page and continue to complete login and data submission.

[0091] (2) Case 2: Upgrading the inventory management page of an e-commerce platform Problem description: The e-commerce platform has upgraded the inventory management page, and the HTML structure of the product list and inventory button has changed. The administrator needs to continue to use automated tools to generate inventory reports. Traditional automated tool problems: The old script is invalid, the inventory button and product list cannot be found, and the operation fails.

[0092] The module's adaptability to cyclic updates: The module finds that the structure of the product list and the ID of the inventory button have changed, automatically re-parses the page DOM, and adjusts the operation instructions to adapt to the new structure. The system also uses the operation history to identify the usage pattern of the inventory button in multiple pages and quickly updates all related operation steps.

[0093] Effect: After the page is upgraded, the administrator can still use the automated system to generate inventory reports without manual intervention.

[0094] (3) Case 3: Update of the report generation page of the online education platform Problem description: The online education platform has updated the layout and functions of the student report generation page. The original report generation buttons and fields have been rearranged. Teachers hope to continue to generate reports in batches through automation tools. Traditional automation tool problem: After the page is upgraded, the original operation script cannot be run, resulting in interrupted operations.

[0095] Adaptability of the cyclic update module: When the system detects changes in the report generation button and field position, the module automatically re-parses the new page layout and updates the operation instructions. By analyzing the operation history, the system quickly adjusts the operation process of generating and sending reports to ensure that the batch report generation and sending functions continue to be effective.

[0096] Effect: Teachers can continue to generate and send reports through simple instructions, and the system automatically adapts to page changes to ensure continuity of operation.

[0097] (1). Continuous and effective automated operation: By monitoring and adapting to changes in the structure of web pages in real time, the system can maintain the continued effectiveness of automated operations. Even if the page undergoes major structural adjustments, the system can still automatically parse and adjust the operation logic, reducing the need for user intervention.

[0098] (2) Reduce operational failures and interruptions: The cyclic update and adaptation module can quickly respond to changes in web pages and reduce the failure or interruption of operations caused by page updates. Even if the DOM structure of the page changes significantly, the system can quickly adjust to ensure the successful execution of the task.

[0099] (3) Improve the stability of long-term use: The system gradually optimizes the operation logic and adapts to the changes of different web pages through operation history recording and learning. This makes the system more stable and adaptable in the long-term use process, and reduces the user's need for script maintenance.

[0100] (4) Reduce maintenance costs: Since the system can automatically adapt to changes in web pages, users do not need to frequently maintain or update automated scripts, which reduces maintenance costs. Especially when web pages are frequently updated, this module can significantly reduce the need for manual intervention.

[0101] The "Cyclic Update and Adaptation Module" ensures the continued effectiveness and long-term stability of the automation process by monitoring web page changes in real time, automatically updating the operation logic, and optimizing the operation path based on the operation history. This module significantly reduces the operation failures and interruptions caused by web page updates, greatly improves the system's adaptability in a dynamic web page environment, and reduces users' maintenance costs.

[0102] 6. Reduce maintenance costs: Since the system can dynamically adapt to changes in web pages, users do not need to frequently maintain and update operation scripts. This reduces maintenance costs and alleviates the technical burden on users.

[0103] Specific instructions for reducing maintenance costs Reducing maintenance costs is one of the important beneficial effects of the "cyclic update and adaptation module" in the present invention. Since the system can dynamically adapt to changes in web pages, users do not need to frequently maintain or update operation scripts, which reduces maintenance costs and alleviates the technical burden of users. This module reduces the reliance on manual update scripts by automatically identifying and processing changes in web pages. The following will analyze how to reduce maintenance costs by dynamically adapting to changes in web pages, and explain in detail when and under what circumstances the system is maintained and updated.

[0104] Principles and case analysis of dynamically adapting to web page changes Web page change types and adaptation mechanisms: (1). Small web page updates: Characteristics of changes: These updates usually involve minor DOM structural changes, such as changes in form field IDs, button position adjustments, CSS style updates, etc. Such changes will not have a significant impact on the overall page layout, but may affect the positioning of specific elements involved in automated operations. Adaptation mechanism: By monitoring these minor changes, the system can automatically adjust the operation instructions without human intervention. For example, if the ID of the username input box on the login page changes from "username" to "user_name", the system will automatically re-parse the page and update the operation instructions to adapt to the new field name. Case: In the internal system of the enterprise, the IT team performs small-scale page maintenance every month and adjusts the naming rules of form fields. Through automation tools, employees can continue to use the same operation instructions to submit weekly reports without the IT team having to rewrite scripts.

[0105] (2) Medium-sized web page updates: Characteristics of changes: These updates usually involve a redesign of the page layout, or an increase or decrease in the number of form fields. For example, adding verification steps, adding new mandatory items, adjusting element positions, etc. Although the page functionality remains unchanged, changes in the DOM structure may cause automated operations to fail. Adaptation mechanism: The system automatically adapts to the new page structure by re-parsing the entire DOM tree to ensure the continuity of the operation process. For example, after adding a verification step, the system automatically identifies the new form element and inserts an additional verification step into the operation process. Case: The online education platform upgraded the student report page and added a security verification field. In traditional scripting tools, administrators need to manually update the script and add verification operations. In this system, the automation tool can dynamically adapt to the new fields and continue to generate and send reports in batches, avoiding the need to manually modify the script.

[0106] (3) Major web page updates: Characteristics of changes: This type of update includes major adjustments to the page architecture, and even re-planning of the entire UI / UX design, affecting the names, positions, and hierarchies of most page elements. For example, after the system is upgraded to a new version, the entire login process or navigation structure may be redesigned. Adaptation mechanism: Although large updates will significantly change the page layout, the system is able to regenerate the operation logic that adapts to the new structure through deep analysis and historical operation records. In this case, the system may require more processing time to adapt to the new page structure, but it still avoids the tedious process of manually writing scripts. Case: An e-commerce platform has significantly updated the backend inventory management system and redesigned the layout of the inventory page. The system automatically adjusts the inventory inspection and report generation process by analyzing the new DOM structure and using the operation history, avoiding the complexity of manually adjusting the script.

[0107] When maintenance and updates are performed: (1) When a web page change is detected automatically: The system monitors the page structure in real time through the DOM monitoring mechanism. If the ID, name, or position of a page element (such as a form field or button) is found to have changed, the system will automatically reparse the page and generate new operation instructions. In this case, maintenance and updates are automatically completed in the background without user intervention. Scenario: For example, if the ID of a form field changes from "email" to "user_email", the system will immediately sense the change and automatically adjust the input process to ensure that the operation is not affected.

[0108] (2) When the operation fails or is abnormal: If certain operations (such as clicking a button or filling out a form) fail during execution, the system automatically performs feedback analysis and tries to reparse the page to find the new structure. If multiple attempts fail, the system may issue a prompt requiring the user to intervene manually or enter specific parameters. Scenario: When checking inventory on an e-commerce platform, if the button position changes and the operation fails, the system will rescan the page and automatically update the click position. If multiple retries still fail, the system will prompt the administrator to manually check the changes to the web page.

[0109] (3) When periodically updating the DOM: The system can be set to regularly recheck and update the DOM structure of the web page, especially on pages that users frequently operate. This strategy is suitable for websites that are updated regularly to ensure that the system always maintains the latest operation logic. Scenario: On an online education platform, the system can regularly check changes to the student report generation page every week. Even if no abnormal situation is triggered, the operation logic can be actively updated to ensure that subsequent report generation tasks will not be interrupted.

[0110] Benefits and details of reduced maintenance costs: (1). Reduce the need to frequently update scripts: In traditional automation tools, whenever a web page changes, users need to manually modify the script, adjust element positioning and operation logic. Through the cyclic update and adaptation module, the system can automatically adapt to changes in the page, greatly reducing the need for users to manually maintain scripts.

[0111] (2)Reducing the workload of technical personnel: - Traditional scripting tools often require users with technical backgrounds to write and maintain scripts. As web pages are constantly updated, technicians need to frequently adjust and test them. This invention reduces the workload of technicians by automatically adapting to web page changes, allowing non-technical personnel to easily use automated tools.

[0112] (3) Long-term stability: The system's dynamic update capability not only reduces the workload of short-term maintenance, but also improves long-term stability. Even if the webpage is frequently updated, the system can still automatically adapt to ensure the continued effectiveness of the operation process. This long-term stability significantly reduces maintenance costs and reduces additional costs caused by system interruptions or operation failures.

[0113] (4) Extend the life cycle of automation scripts: In traditional automation tools, the effective life cycle of scripts is usually short, and as web pages continue to evolve, scripts need to be constantly updated. However, the present invention can extend the effective life cycle of scripts through dynamic parsing and historical learning, reducing the need for frequent replacement and maintenance of scripts.

[0114] Maintenance cost analysis based on actual cases Case 1: Adjustment of the internal system page of the enterprise Traditional maintenance costs: Every time a minor adjustment is made to an internal system page, technicians need to rewrite the script and test it to ensure that the automated task is executed smoothly. Effect of reducing maintenance costs: The system automatically adapts to minor adjustments to web pages, such as changes in form field names or button positions, reducing the cost of manual script maintenance after each page update. Each update saves 2-3 hours of manual maintenance time for technicians.

[0115] (2) Case 2: E-commerce platform inventory management system upgrade Traditional maintenance costs: After a major upgrade, administrators need to manually update the inventory management script, which may take several days of testing and adjustment. Effect of reducing maintenance costs: Through the cyclic update module, the system can automatically adapt to changes in page structure, reducing the workload of administrators to manually adjust scripts. The entire adjustment process is shortened from the original 3 days to a few hours, greatly reducing maintenance costs.

[0116] (3) Case 3: Dynamic changes in online education platforms Traditional maintenance costs: Online education platforms are frequently upgraded, and teachers need to rely on technical staff to maintain automated operation scripts, especially in generating and sending reports. Effect of reducing maintenance costs: The automated system can automatically parse the new page structure after the platform changes, reducing the need for teachers to rely on technical support and ensuring that report generation tasks are not interrupted, saving hours of maintenance time for each update.

[0117] By combining automatic monitoring of web page changes, periodic updates, and failure feedback processing, the "cyclic update and adaptation module" of the present invention significantly reduces the need for users to manually maintain and update operation scripts, reduces the workload of technicians, extends the effective life cycle of automation scripts, and improves the long-term stability of the system. Through the analysis of actual cases, it is proved that the module has the advantage of effectively reducing maintenance costs in different web page environments.

[0118] 7. Enhance system intelligence: Using advanced natural language processing technology, the system can intelligently parse and execute user instructions, making browser automation more intelligent and efficient. Improve the intelligence level of the system and user experience. The present invention solves the deficiencies in the prior art through innovative technical solutions and device designs, and provides an efficient, intelligent, and user-friendly browser automation solution. There are too many slogans, and the so-called improvement and reduction in this article do not provide any specific cases and effect analysis.

[0119] Glossary: Provide a definition or brief explanation of the technical keywords in the proposal. Abbreviations need to be given in full in Chinese and English. For the same object being described, please ensure that the terms used in the briefing are consistent.

[0120] Example: SRv6 SFC (Segment Routing over IPv6 Service Function Chain): is a technology that adds SRv6 path information to the original message to guide the message to pass through the application layer service device in sequence along the specified path.

[0121] SC (Service Classifier): Located at the edge of the SRV6 SFC service chain network, it is the source node of the service chain path. SC can use different diversion methods to introduce service data into the SRV6 TE Policy tunnel for forwarding.< / header> < / button>

Claims

1. A browser automation method based on natural language processing, specifically comprising the following steps: a user inputs a natural language instruction, and the user describes the operation that the user wants to perform in natural language; natural language processing parses the instruction, identifies the intention, and generates operation steps; web page DOM parsing extracts the web page structure and interactive elements to provide an operation basis; and interactive operations are performed with web page elements according to the generated operation steps; Monitor operation status in real time, handle exceptions, and adjust strategies; Update the model based on web page changes; Web page DOM parsing: Get the complete HTML structure of a web page, parse and simplify it, extract elements related to user interaction, such as buttons, input boxes, links; parse the HTML document of a web page, including static content and dynamically loaded content; a. Parsing of static content Get the HTML document of the web page directly through HTTP request; This is the basic way to parse static content. Use tools such as curl, requests, and BeautifulSoup to grab and parse HTML content. Use the parser BeautifulSoup or lxml to convert the obtained HTML document into a DOM tree and extract all elements in it, including nested structures. In the DOM tree, extract the attributes of each node, such as id, class, name, href, and src, to provide positioning information for subsequent operations. b. Parsing of dynamic content Modern web pages often contain dynamic content loaded via JavaScript or AJAX; use browser automation tools such as Selenium or Puppeteer to simulate user behavior and set wait conditions WebDriverWait or waitForSelector to wait for dynamic elements in the page to load; directly execute the JavaScript code of the page, or use interfaces such as fetch and XMLHttpRequest in the window object to obtain dynamically loaded data; in single-page applications, page content is loaded asynchronously through routing and components; identify changes in page content by monitoring the pushState, popState, or hashchange events of the browser's history object; Use MutationObserver to monitor DOM changes to ensure that elements dynamically inserted via JavaScript are captured.

2. The browser automation method based on natural language processing according to claim 1, characterized in that The web page DOM parsing simultaneously performs web page DOM simplification, uses natural language processing (NLP) technology to perform semantic analysis on the text content in the web page, identifies the content related to the user instruction and forms a DOM tree, and the DOM tree is simplified to prioritize the retention of content.

3. The system of the browser automation method based on natural language processing according to claim 1, characterized in that It includes a natural language instruction parsing module, a web page DOM parsing module, an operation execution module, a state monitoring and feedback module, and a loop update and adaptation module; the natural language instruction parsing module uses the natural language processing technology GPT-4 to parse the natural language instructions input by the user; the module can recognize and understand the user's intention and convert the natural language instructions into specific operation steps; the module includes functions such as instruction parsing, intention recognition, and command generation; the web page DOM parsing module obtains and parses the HTML structure of the web page and converts it into a simplified DOM tree; This module extracts elements related to user interaction, such as buttons, input boxes, and links, to provide basic data for operation execution; the operation execution module performs interactive operations on web page elements according to the parsed operation steps, including clicking buttons, setting input box values, submitting forms, etc., and the module is implemented by directly interacting with DOM elements; the status monitoring and feedback module monitors the page status in real time and evaluates the completion of the operation; the operation strategy is dynamically adjusted according to the feedback information to deal with possible abnormal situations or errors; The cyclic update and adaptation module updates the natural language processing model according to the new DOM structure and operation history; The system can dynamically adapt to changes in web page content and structure and continue to perform subsequent operations.

Images