An ai-based cross-system web automation operation method and system

Abstract

This invention provides an AI-based cross-system web automation operation method and system. The method includes: registering all web system functions within an enterprise to an AI system, inputting the web operation flow and key information extraction rules for each function; integrating AI capabilities into web applications as plug-ins; when a user selects target information, the AI ​​recognizes the user's intent, generates a shortcut function entry in the browser's right-click menu, and extracts key information; automatically executing corresponding cross-system web operations based on the automation capabilities of Playwright and Browser Use; providing auxiliary enhancement services through floating windows; and allowing users to drive automated web operations via natural language commands. The system comprises six modules, including a function registration module and an AI integration module, to implement the above method. This invention solves the problems of cumbersome web operations, difficult cross-system collaboration, and low automation levels in existing web applications. It eliminates the need for manual operation, improves operational efficiency, achieves intelligent collaboration among multiple systems, and provides convenient user access.

Description

Technical Field

[0001] This invention relates to the field of web automation technology and artificial intelligence applications, specifically to an AI-based cross-system web automation operation method and system. Background Technology

[0002] As enterprises advance their digital transformation, they typically deploy multiple independent web systems, each undertaking different business functions, such as vehicle trajectory query systems and vehicle alarm systems. Currently, when using these web systems, enterprise users need to manually complete a series of operations such as page navigation, information input, and button clicks, which is cumbersome, time-consuming, and labor-intensive.

[0003] While some technologies exist related to speech recognition and web automation, these technologies are relatively outdated. Speech recognition accuracy is low, and the adaptability of web automation is poor, failing to achieve accurate conversion between natural language and web operations. Furthermore, the various web systems within an enterprise operate independently, lacking effective collaboration mechanisms. When users operate across systems, they need to manually switch systems and repeatedly input key information (such as license plates and dates), resulting in extremely poor operational convenience and severely impacting work efficiency. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the purpose of this invention is to provide an AI-based cross-system web automation operation method and system, solving the problems of existing technologies where web operations require manual completion, are cumbersome and inefficient, and where web systems are independent and lack collaborative mechanisms.

[0005] To achieve the above-mentioned technical effects, the present invention adopts the following technical solution:

[0006] According to a first aspect of the present invention, an AI-based cross-system web automation operation method is provided, comprising the following steps:

[0007] Step 1: Register the functions of all web systems within the enterprise to the AI ​​system. Each system function corresponds to a complete web operation process, which includes page navigation path, input operation, click operation, and optional AI-assisted enhancement operation. When registering the system function, clearly enter the preset key information extraction rules for each function. The key information is the core parameters required to execute the system function.

[0008] Step 2: Integrate the AI ​​system into the web application as a plug-in to enable real-time data interaction between the AI ​​system and various web systems. The AI ​​system can obtain the content of the current web page and user operation behavior in real time.

[0009] Step 3: When a user selects target information in any web system, the AI ​​system automatically recognizes the user's operation intention, generates the corresponding shortcut function entry in the browser's right-click menu based on the system functions already registered within the enterprise, and automatically extracts key information from the current target information;

[0010] Step 4: The AI ​​system calls the corresponding registered system function based on the shortcut function entry selected by the user. Based on the WEB automation function provided by Playwright and Browser Use, it automatically executes the complete WEB operation process corresponding to the system function without requiring any other operation from the user.

[0011] Step 5: The AI ​​system automatically identifies the current user's operational intent and generates a function introduction, usage guide, or page information summary for the current page through a floating window; at the same time, after the system function is executed, it generates a corresponding result report based on the preset AI-assisted enhanced operation;

[0012] Step 6: The user describes the operation task through natural language or the WEB AI shortcut. The AI ​​big model converts the natural language description into an executable WEB operation process, and then automatically executes the operation process based on the WEB automation functions provided by Playwright and Browser Use, without requiring any other operation from the user.

[0013] Optionally, in step 1, the function registration step of the WEB system adopts a visual interface, which supports the modification, deletion and query of system function information. The system function information includes system name, page navigation path, WEB operation process, AI-assisted enhancement operation and key information extraction rules.

[0014] Optionally, in step 2, the AI ​​system is integrated as a browser plugin, which can collect DOM elements, page display information and user operation events of the current browser page in real time, and transmit the collected data to the AI ​​system in real time.

[0015] Optionally, in step 3, the user intent recognition is achieved through keyword extraction and semantic analysis, the shortcut function entry corresponds one-to-one with the registered system functions, and the extraction of key information is completed based on preset key information extraction rules.

[0016] Optionally, in step 6, the natural language processing process includes: when inputting voice, converting the voice signal into a text command through a voice recognition interface; and the AI ​​big model performing semantic understanding and task decomposition on the text command, converting it into an executable WEB operation process.

[0017] According to a second aspect of the present invention, an AI-based cross-system web automation operating system is provided for implementing the above-described method, the system comprising:

[0018] Function registration module: Used to receive function registration information of all WEB systems within the enterprise, store the WEB operation process and key information extraction rules corresponding to the functions of each system, the WEB operation process includes page navigation path, input operation, click operation and optional AI-assisted enhancement operation;

[0019] AI Integration Module: Integrated into web applications as a plug-in to enable real-time data interaction between the AI ​​system and various web systems, and to obtain the current web page content and user operation behavior;

[0020] Intent Recognition and Quick Function Generation Module: Used to recognize the user's operational intent on the web page, generate quick function entries in the browser's right-click menu based on the system functions stored in the function registration module, and automatically extract key information from the current page;

[0021] WEB Automation Execution Module: Based on the WEB automation functions provided by Playwright and Browser Use, it receives operation instructions sent by the AI ​​system and automatically executes the corresponding WEB operation process to achieve cross-system automated operation.

[0022] AI-assisted enhancement module: used to identify the user's current operation intent and provide page function introductions, usage guidance or information summaries in the form of a floating window; at the same time, it generates operation result reports based on preset AI-assisted enhancement operations;

[0023] Natural Language Processing Module: This module receives natural language commands (voice or text) from the user, converts them into executable web operation processes using a large AI model, and sends them to the web automation execution module for execution.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. No manual operation of web operations is required. AI converts natural language into executable web operations. Combined with the automation capabilities of Playwright and Browser Use, the web operations are fully automated, greatly reducing user operation costs and improving operation efficiency.

[0026] 2. Integrate AI capabilities into web applications as plugins, allowing users to directly use AI-assisted functions (such as information summarization and context filling) on ​​web pages. This is convenient and does not require switching to other AI tools.

[0027] 3. It can automatically recognize the user's operation intent, generate a shortcut function entry in the browser right-click, automatically extract key information, avoid users from repetitive input, and further improve the convenience of operation;

[0028] 4. Enable intelligent collaboration among multiple web systems within an enterprise, integrate independent system functions, and use AI as a bridge for cross-system automation and multi-system function integration to solve the problem of cumbersome cross-system operations;

[0029] 5. Provides AI-assisted enhancement features, automatically generating page function introductions, usage guides, and information summaries to help users quickly familiarize themselves with system functions, obtain key information, and further improve work efficiency. Attached Figure Description

[0030] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0031] Figure 1 This is a flowchart of each step in the AI-based cross-system web automation operation method described in Example 1. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0033] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0034] It should be noted that similar reference numerals and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, all directional indications (such as up, down, left, right, front, back, bottom, etc.) in this application are only used to explain the relative positional relationships and movements between components in a specific orientation (as shown in the figures). If the specific orientation changes, the directional indication will also change accordingly. Furthermore, descriptions involving "first," "second," etc., in this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated.

[0035] Example 1: AI-based cross-system web automation operation method

[0036] This embodiment provides an AI-based cross-system web automation operation method, combined with Figure 1 The flowchart shown below details the specific steps:

[0037] Step 1: System Function Registration: Register the functions of all web systems within the enterprise to the AI ​​system. Each system function corresponds to a complete web operation flow, including page navigation path, input operations, click operations, and optional AI-assisted enhancement operations. During system function registration, clearly define and input the preset key information extraction rules for each function. The key information consists of core parameters required to execute the system function, such as license plate number, date, alarm time, etc.

[0038] This embodiment takes a vehicle trajectory query system (System A) and a vehicle alarm system (System B) as examples:

[0039] System A's registration information: The page navigation path is https: / / demo1, and the web operation process is as follows: 1. Open the page corresponding to this navigation path; 2. Find the license plate number input box and enter the license plate information; 3. Find the trajectory date input box and enter the date information; 4. Click the query button; AI-assisted enhanced operation is: automatically generate a vehicle trajectory summary report based on the map trajectory displayed on the page; the key information extraction rule is: extract "license plate" and "date" from the current page or the information selected by the user.

[0040] System B's functional registration information: The page navigation path is https: / / demo2, and the WEB operation process is as follows: 1. Open the page corresponding to this navigation path; 2. Click the vehicle alarm list in the upper right corner of the page; 3. Display vehicle license plate, alarm time, and other alarm information; The key information extraction rule is: extract "license plate" and "alarm time" from the alarm list.

[0041] Step 2: AI Capability Integration

[0042] The AI ​​system is integrated into the user's web browser as a browser plugin. After the plugin is installed, it automatically establishes a data connection with system A and system B within the enterprise. It can collect DOM elements, page display information and user operation events of the current browser page in real time and transmit the collected data to the AI ​​system in real time.

[0043] Step 3: User Intent Recognition and Quick Action Generation

[0044] When a user selects target information in any web system, the AI ​​system automatically recognizes the user's intention and generates a corresponding shortcut function entry in the browser's right-click menu based on the system functions already registered within the enterprise. It also automatically extracts key information from the current target information. User intention recognition is achieved through keyword extraction and semantic analysis. The shortcut function entry corresponds one-to-one with the registered system functions, and the extraction of key information is completed based on preset key information extraction rules.

[0045] For example, when a user opens the page of System B (https: / / demo2) and clicks the vehicle alarm list in the upper right corner, the list displays alarm information for multiple vehicles. When the user selects the alarm information of one of the vehicles (including the license plate and alarm time) with the mouse, the AI ​​system obtains the user's selection operation and the current page content through the AI ​​integration module, and automatically recognizes the user's intention as "to query the trajectory information of this vehicle". Subsequently, based on the registered function of System A, the AI ​​system generates a shortcut function entry for "query the trajectory of this vehicle" in the browser's right-click menu, and automatically extracts key information from the selected alarm information, such as the license plate and date.

[0046] Step 4: Cross-system automated execution. The AI ​​system calls the corresponding registered system function based on the shortcut function entry selected by the user. Based on the WEB automation function provided by Playwright and Browser Use, it automatically executes the complete WEB operation process corresponding to the system function without requiring any other operation from the user.

[0047] Specifically, after the user clicks the "Query Vehicle Track" shortcut in the right-click menu, the AI ​​system sends an operation command to the WEB automation execution module. Based on the WEB automation functions provided by Playwright and Browser Use, the WEB automation execution module automatically executes the vehicle track query process of System A: 1. Automatically open the page of System A (https: / / demo1); 2. Automatically locate the license plate number input box and enter the extracted license plate number; 3. Automatically locate the track date input box and enter the extracted date; 4. Automatically click the query button to complete the track query operation, without requiring manual intervention from the user.

[0048] Step 5: AI-assisted enhancement. The AI ​​system automatically identifies the current user's operational intent and generates a function introduction, usage guide, or page information summary for the current page through a floating window. At the same time, after the system function is executed, a corresponding result report is generated based on the preset AI-assisted enhancement operation.

[0049] Specifically, during the trajectory query operation performed by System A, the AI-assisted enhancement module automatically identifies the user's current operation intent as "query vehicle trajectory" and displays operation guidance in the upper right corner of the page via a floating window (such as "Automatically querying vehicle trajectory, please wait"). After the trajectory query is completed and the map trajectory is displayed on the page, the AI-assisted enhancement module automatically generates a vehicle trajectory summary report (including trajectory start and end points, travel time, key nodes, etc.) and displays it through the floating window. At the same time, the floating window also provides an introduction to the functions of System A (such as "This page can query historical vehicle trajectories and supports exporting trajectory reports"), making it convenient for users to quickly understand the system's functions.

[0050] Step 6: Natural Language Driven Operation. Users describe the operation task through natural language or the WEB AI shortcut. The AI ​​big model transforms the natural language description into an executable WEB operation process. Based on the WEB automation functions provided by Playwright and Browser Use, the operation process is executed automatically without any other operation from the user.

[0051] Users can input the text command "Query the trajectory of license plate ABC123 on 2026-03-18 and generate a summary report" through the natural language input box of the browser plugin. After receiving the command, the natural language processing module uses an AI model to convert it into an executable web operation process: 1. Open system A page https: / / demo1; 2. Enter the license plate and date; 3. Click the query button; 4. Generate a trajectory summary report. Subsequently, the web automation execution module automatically executes the operation process. After the operation is completed, the trajectory summary report is sent to the location specified by the user (such as a browser pop-up or email).

[0052] Example 2: AI-based cross-system web automation operating system

[0053] This embodiment provides an AI-based cross-system web automation operating system for implementing the method described in Embodiment 1. The system includes a function registration module, an AI integration module, an intent recognition and quick generation module, a web automation execution module, an AI-assisted enhancement module, and a natural language processing module. The specific functions of each module are as follows:

[0054] 1. Function Registration Module: This module features a visual interface design, allowing enterprise administrators to input functional information for various web systems, including system name, page navigation path, web operation process (broken down into specific steps such as navigation, input, and clicking), AI-assisted enhanced operation, and key information extraction rules. The module 1 categorizes and stores the input information, establishing a correspondence between "system function - operation process - key information" for other modules to access. It also supports modification, deletion, and querying of functional information, facilitating timely adjustments by enterprises based on system updates.

[0055] 2. AI Integration Module: Implemented as a browser plugin (supporting mainstream browsers such as Chrome and Edge). The plugin starts automatically after installation and establishes HTTP communication connections with various web systems within the enterprise. It collects DOM elements, page display information, and user operation events (such as mouse clicks, selections, and keyboard inputs) of the current browser page in real time and transmits the collected data to the intent recognition and quick generation module and the AI-assisted enhancement module in real time.

[0056] 3. Intent Recognition and Quick Generation Module: This module has a built-in trained user intent recognition model. It receives page data and user operation events from the AI ​​integration module and identifies the user's operation intent through keyword extraction and semantic analysis. Simultaneously, it calls the system function information stored in the function registration module to match the system function corresponding to the user's intent, generating a corresponding quick function entry in the browser's right-click menu. Furthermore, this module automatically extracts key parameters (such as license plate number and date) from the current page or user-selected information according to preset key information extraction rules and stores them in a temporary database for use by the WEB automation execution module.

[0057] 4. Web Automation Execution Module: Integrates the web automation interfaces of Playwright and Browser Use, supporting multi-browser adaptation; this module receives operation instructions (including target system functions and key parameters) sent by the intent recognition and quick generation module or the natural language processing module, calls the corresponding web operation flow in function registration module 1, automatically parses the operation steps, and simulates user operations (such as opening a page, entering information, and clicking a button) through the automation interface to achieve full automation of web operations; during the operation, it provides real-time feedback on the operation status (such as "navigating to the target page" or "input completed") to the AI-assisted enhancement module.

[0058] 5. AI-Assisted Enhancement Module: Receives operation status feedback from the WEB Automation Execution Module, page data from the AI ​​Integration Module, and user intent from the Intent Recognition and Quick Generation Module. It then displays relevant information to the user via a floating window. The floating window content can be dynamically adjusted based on the user's intent, including page function introductions, operation guidance, a summary of current page information, and operation result reports. The operation result reports are automatically generated by the AI ​​model based on the results displayed on the page and can be exported to PDF, Excel, and other formats.

[0059] 6. Natural Language Processing Module: Integrates a speech recognition interface and large AI models (such as the GPT series and ByteDance Seed model), supporting users to input operation commands via voice or text. When inputting by voice, the speech signal is converted into a text command through the speech recognition interface. Subsequently, the large AI model performs semantic understanding and task decomposition on the text command, converting it into an executable WEB operation process (consistent with the operation process format in the function registration module), and sends it to the WEB automation execution module for execution. At the same time, this module supports context association and can optimize the command parsing accuracy based on the user's historical operation records.

[0060] In this embodiment, the modules work together to realize cross-system WEB automated operation driven by natural language and AI-assisted enhancement functions, effectively solving the problems of cumbersome operation, difficulty in cross-system collaboration and low degree of automation in the prior art, and significantly improving user operation efficiency and user experience.

[0061] The specific embodiments of the present invention have been described above. Based on the above description, those skilled in the art can make various changes and modifications without departing from the technical concept of the present invention.

Claims

1. A cross-system web automation operation method based on AI, characterized in that, Includes the following steps: Step 1: Register the functions of all web systems within the enterprise to the AI ​​system. Each system function corresponds to a complete web operation process, which includes page navigation path, input operation, click operation, and optional AI-assisted enhancement operation. When registering the system function, clearly enter the preset key information extraction rules for each function. The key information is the core parameters required to execute the system function. Step 2: Integrate the AI ​​system into the web application as a plug-in to enable real-time data interaction between the AI ​​system and various web systems. The AI ​​system can obtain the content of the current web page and user operation behavior in real time. Step 3: When a user selects target information in any web system, the AI ​​system automatically recognizes the user's operation intention, generates the corresponding shortcut function entry in the browser's right-click menu based on the system functions already registered within the enterprise, and automatically extracts key information from the current target information; Step 4: The AI ​​system calls the corresponding registered system function based on the shortcut function entry selected by the user. Based on the WEB automation function provided by Playwright and Browser Use, it automatically executes the complete WEB operation process corresponding to the system function without requiring any other operation from the user. Step 5: The AI ​​system automatically identifies the current user's operational intent and generates a function introduction, usage guide, or page information summary for the current page through a floating window; at the same time, after the system function is executed, it generates a corresponding result report based on the preset AI-assisted enhanced operation; Step 6: The user describes the operation task through natural language or the WEB AI shortcut. The AI ​​big model converts the natural language description into an executable WEB operation process, and then automatically executes the operation process based on the WEB automation functions provided by Playwright and Browser Use, without requiring any other operation from the user.

2. The AI-based cross-system web automation operation method according to claim 1, characterized in that, In step 1, the function registration step of the WEB system adopts a visual interface, which supports the modification, deletion and query of system function information. The system function information includes system name, page navigation path, WEB operation process, AI-assisted enhancement operation and key information extraction rules.

3. The AI-based cross-system web automation operation method according to claim 1, characterized in that, In step 2, the AI ​​system is integrated as a browser plugin, which can collect DOM elements, page display information and user operation events of the current browser page in real time, and transmit the collected data to the AI ​​system in real time.

4. The AI-based cross-system web automation operation method according to claim 1, characterized in that, In step 3, the user intent recognition is achieved through keyword extraction and semantic analysis. The shortcut function entry corresponds one-to-one with the registered system functions. The extraction of key information is completed based on preset key information extraction rules.

5. The AI-based cross-system web automation operation method according to claim 1, characterized in that, In step 6, the natural language processing process includes: when voice input occurs, the voice signal is converted into a text command through a voice recognition interface; the AI ​​big model performs semantic understanding and task decomposition on the text command, and converts it into an executable WEB operation process.

6. An AI-based cross-system web automation operating system, characterized in that, The system for implementing the method according to any one of claims 1-5, the system comprising: Function registration module: Used to receive function registration information of all WEB systems within the enterprise, store the WEB operation process and key information extraction rules corresponding to the functions of each system, the WEB operation process includes page navigation path, input operation, click operation and optional AI-assisted enhancement operation; AI Integration Module: Integrated into web applications as a plug-in to enable real-time data interaction between the AI ​​system and various web systems, and to obtain the current web page content and user operation behavior; Intent Recognition and Quick Function Generation Module: Used to recognize the user's operational intent on the web page, generate quick function entries in the browser's right-click menu based on the system functions stored in the function registration module, and automatically extract key information from the current page; WEB Automation Execution Module: Based on the WEB automation functions provided by Playwright and Browser Use, it receives operation instructions sent by the AI ​​system and automatically executes the corresponding WEB operation process to achieve cross-system automated operation. AI-assisted enhancement module: used to identify the user's current operation intent and provide page function introductions, usage guidance or information summaries in the form of a floating window; at the same time, it generates operation result reports based on preset AI-assisted enhancement operations; Natural Language Processing Module: This module receives natural language commands input by the user, transforms them into executable web operation processes using a large AI model, and sends them to the web automation execution module for execution.

Images