Webpage form control intelligent filling method and system based on page change monitoring

Abstract

The invention discloses a webpage form control intelligent filling method and system based on page change monitoring, and the method comprises the steps: confirming a webpage form control, and adding an identifier for a node; according to the form control, generating simplified node structure data; setting a cyclic variable; the method comprises the following steps: monitoring the change of an HTML (Hypertext Markup Language) structure of a page by adopting a Mutation Observer; searching a target node according to the target node identifier and executing an operation; ending monitoring and acquiring a monitoring result; processing the monitoring result to generate difference data; performing intelligent analysis and decision making according to the difference data and the node structure data; and automatically filling in a webpage form control according to a feedback result of the large language model. According to the method, the viewport limitation problem is solved, the processing efficiency is improved by 60% or above, the compatibility of form controls of various Web UI component libraries reaches 95% or above, and the method has the advantages of being high in universality, low in cost and high in accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of webpage automation, in particular to a webpage form control intelligent filling method and system based on page change monitoring. BACKGROUND

[0002] With the rapid development of Internet applications, the demand for webpage form filling automation is growing. Existing webpage form automatic filling methods include: 1) Rule matching-based filling method: identify form controls and fill them in by pre-set field names, CSS selectors, etc.

[0003] 2) Image recognition-based filling method: use optical character recognition technology to identify form controls and labels on the page, and then perform filling operations through image positioning.

[0004] 3) DOM operation-based filling method: directly manipulate page DOM elements for filling, find target controls by parsing page structure, and then simulate user operations to complete filling.

[0005] 4) Multi-modal large language model-based filling method: use a large language model that can handle images and text simultaneously to fill out forms, analyze the combination of page screenshots and text descriptions, intelligently identify form controls, and generate operation instructions.

[0006] The above-mentioned form automatic filling methods have the following technical defects: Viewport restriction problem: image recognition and multi-modal model-based methods can only handle currently visible page elements, and cannot handle form controls that need to be scrolled, expanded, or triggered to display. This limitation makes it difficult to support many practical application scenarios effectively.

[0007] Complex control compatibility problem: for UI component library form controls widely used in modern web applications (such as AntDesign, Element UI, etc.), the success rate of existing methods is low, especially for controls that require multiple-step interactions such as cascading selectors, date and time selectors, etc.

[0008] Cost and efficiency problem: multi-modal large language models are powerful but have high computational costs and slow response speeds, with a single processing fee of 0.2-0.5 yuan RMB, making them unsuitable for large-scale applications. Meanwhile, processing time can take up to 15-30 seconds, making it difficult to meet real-time requirements.

[0009] Accuracy and reliability problem: lack of effective result verification mechanism, unable to ensure the accuracy and completeness of filling operations. When abnormal situations occur during filling, existing methods have difficulty in effectively detecting and recovering errors.

[0010] Technical universality problem: existing methods are often designed for specific types of form controls, which are difficult to adapt to changing web technologies and emerging interaction modes, and have poor scalability and maintainability. SUMMARY

[0011] In order to solve the defects of the prior art, the purpose of the present application is to provide a web page form control intelligent filling method and system based on page change monitoring, to improve the response speed and accuracy of web form filling.

[0012] To achieve the above purpose, the web page form control intelligent filling method based on page change monitoring provided by the present application comprises: Confirming the web form control, adding an identifier to the node; Generating a simplified node structure data according to the form control; Setting a loop variable; Using MutationObserver to monitor the HTML structure changes of the page; Finding the target node according to the target node identifier and performing the operation; End the monitoring and get the monitoring result; Processing the monitoring result to generate difference data; Intelligently analyzing and deciding according to the difference data and the node structure data; According to the large language model feedback result, automatically filling the web form control.

[0013] Further, the step of confirming the web form control and adding an identifier to the node further comprises: Taking the HTML node of the form control as the root node, traversing all descendant nodes, and adding a unique identifier; Check if the HTML node has an ID attribute; if not, automatically generate a unique ID attribute for the node according to the generation rule of prefix identifier plus timestamp plus random number sequence.

[0014] Further, the step of confirming the web form control and adding an identifier to the node further comprises: for the cascading selector component, the node is divided into the following types: cascader type represents the main control container, panel type represents the option panel, option type represents the specific option, input type represents the input box, button type represents the trigger button, and container type represents the normal container. According to the type prefix plus timestamp plus serial number identification rule, add an identifier to the node.

[0015] Further, the step of generating the simplified node structure data according to the form control further comprises: traversing all descendant nodes by using a depth-first traversal algorithm, extracting key technical information of each node, and compressing the complete DOM structure data into lightweight JSON format data to generate the simplified node structure data for the form control.

[0016] Further, the key technical information comprises: node label type, unique identifier, important attribute value, hierarchical relationship position, and current state information, and the style attribute and event attribute are filtered.

[0017] Further, the step of setting the loop variable further comprises: current target node identifier, current operation type, loop counter, maximum loop count limit, and operation history record container, and the operation type comprises: single click, double click, and fill-in.

[0018] Further, the step of using MutationObserver to monitor the HTML structure changes further comprises: The monitoring configuration parameters comprise: childList parameter set to true value to monitor the addition and deletion of child nodes; subtree parameter set to true value to monitor all descendant nodes; attributes parameter set to true value to monitor attribute changes; and attributeOldValue parameter set to true value to record the values before the attribute changes.

[0019] Further, the difference data further comprises: identifier and structure information of the added node, identifier of the deleted node, identifier and change content of the attribute change node, and identifier of the display and hide state change node.

[0020] Further, the step of performing intelligent analysis and decision-making according to the difference data and the node structure data further comprises: constructing a structured data packet of a large language model, wherein the structured data packet of the large language model comprises: current node structure state, page change difference data, target fill-in content information, and historical operation record; calling an API interface of the large language model to perform intelligent analysis and decision-making; obtaining a decision result returned by the model, wherein the decision result comprises: fill-in completion state judgment, next step target node identifier, next step operation type suggestion, and decision confidence score information.

[0021] Further, the method further comprises: for the same domain iframe, directly accessing the internal DOM structure through the contentWindow and contentDocument attributes of the iframe to perform node positioning and operation triggering; For cross-domain iframe, the main page sends operation instructions to the iframe through the postMessage API, the agent script in the iframe receives the instructions and performs corresponding operations, and then returns the execution results to the main page through messages.

[0022] Further, it also includes: the MutationObserver of the main page listens to the changes in the main page range, and the agent script in each iframe listens to the changes in the respective iframe.

[0023] To achieve the above object, the application also provides a web form control intelligent filling system based on page change monitoring, which adopts the web form control intelligent filling method based on page change monitoring as described above to realize the intelligent filling of web form controls.

[0024] Compared with the prior art, the application has the following beneficial effects: It can handle form controls at any position on the page, including controls that need to be displayed by scrolling, dynamically expanded, and displayed in pop-ups, completely solving the fundamental limitation of the prior art that can only handle elements within the current viewport. The success rate of handling form controls outside the viewport is above 92%; Using a simplified node structure data and a text-based large language model, the data transmission volume is reduced by more than 70% compared to the multi-modal solution, significantly improving the processing efficiency; Through the cyclic interaction mechanism and page change monitoring, complex form controls of various modern Web UI component libraries can be effectively handled, with an average success rate of more than 95%, and excellent compatibility with complex controls; Using a text-based large language model instead of a multi-modal model, the computational cost is reduced by more than 80%, significantly reducing the usage cost; Through page change monitoring and multi-round verification mechanism, the execution result of each operation can be detected in real time, and abnormal situations can be found and handled in time to ensure the accuracy and completeness of the filling.

[0025] It does not depend on specific UI frameworks or form control implementation methods, and has good cross-platform and cross-framework adaptability.

[0026] Based on the understanding and reasoning ability of the large language model, it can handle complex form logic and interaction processes, such as conditional display fields, dynamic verification rules, and associated field linkage, etc. advanced scenarios, and has stronger adaptability and intelligence level compared to traditional methods based on rules.

[0027] Other features and advantages of the application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0028] The accompanying drawings are included to provide a further understanding of the present application, and are incorporated in and constitute a part of this specification, illustrate embodiments of the present application and serve to explain the principles of the present application, and should not be taken as limiting of the present application. In the drawings: Figure 1 A flow chart of a method for intelligent filling of a web form control based on page change monitoring according to an embodiment of the present application; Figure 2 A flow chart of a method for intelligent filling of a web form control based on page change monitoring according to an embodiment of the present application; Figure 3 A flow chart of a method for intelligent filling of a web form control based on page change monitoring according to an embodiment of the present application; Figure 4 A flow chart of a method for intelligent filling of a web form control based on page change monitoring according to an embodiment of the present application; Figure 5 A flow chart of a method for intelligent filling of a web form control based on page change monitoring according to an embodiment of the present application; Figure 6 A flow chart of a method for intelligent filling of a web form control based on page change monitoring according to an embodiment of the present application; Figure 7 A flow chart of a method for intelligent filling of a web form control based on page change monitoring according to an embodiment of the present application; Figure 8 A flow chart of a method for intelligent filling of a web form control based on page change monitoring according to an embodiment of the present application; Figure 9 A flow chart of a method for intelligent filling of a web form control based on page change monitoring according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] The preferred embodiments of the present application will be described herein below with reference to the accompanying drawings. It is to be understood that the preferred embodiments described herein are merely illustrative and explanatory of the present application and are not intended to limit the present application in any way.

[0030] Embodiments of the present application will be described below in greater detail with reference to the accompanying drawings. While certain embodiments of the present application are shown in the drawings, it is understood that the present application can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present application will be thoroughly and completely understood. It should be understood that the drawings and embodiments of the present application are for illustrative purposes only and are not intended to limit the scope of the present application.

[0031] As used herein, the term "includes" and its variants are to be read to be analogous to "comprises," or "comprising." The term "based on" is to be read as "based, at least in part, on." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments."

[0032] It should be noted that the terms "first", "second", and the like in the present application can be mentioned only for distinguishing different devices, components or parts, and do not imply the order or interdependence of the functions performed by these devices, components or parts.

[0033] It should be noted that the modification of "one" or "multiple" mentioned in the present application is illustrative but not restrictive, and those skilled in the art should understand that "one or more" should be understood unless otherwise explicitly indicated in the context. "Multiple" should be understood as two or more.

[0034] Embodiment 1 The embodiment of the present application provides a web form control intelligent filling method based on page change monitoring.

[0035] Figure 1 The flowchart of the web form control intelligent filling method based on page change monitoring according to the embodiment 1 of the present application is described in detail below with reference to the web form control intelligent filling method based on page change monitoring of the present application. Figure 1

[0036] First, in step 101, the target form control node is obtained.

[0037] In the embodiment of the present application, the content data to be filled in and the HTML node position of the target form control are determined, the node reference of the target form control is obtained through the DOM query mechanism, and the corresponding relationship between the filling content and the target control is established.

[0038] In step 102, a node identifier is added to the node, and a simplified node structure data is generated In the embodiment of the present application, the HTML node of the form control is taken as the root node, a depth-first traversal algorithm is used to traverse all descendant nodes, and a unique identifier is added to each node. The identifier generation rule is: prefix string + timestamp + random number sequence, which ensures the uniqueness within the page range. In the embodiment of the present application, the depth-first traversal is performed on all descendant nodes, which ensures that each node has a unique identifier.

[0039] ​In the embodiments of the present application, based on the traversal result, the key technical information of each node is extracted to generate simplified node structure data for form controls, including node label type, unique identifier, important attribute value, hierarchical relationship position, current state information, etc. Non-core information such as style attribute and event attribute is filtered out.

[0040] In the embodiments of the present application, by selectively extracting the key technical attributes of the nodes and filtering out non-core information such as style and event, the complete DOM structure data is compressed into lightweight JSON format data. On the premise of ensuring information integrity, the data volume can be reduced by more than 70% compared to the complete DOM structure, significantly reducing data transmission volume and improving the processing efficiency of large language models.

[0041] In step 103, the setting of the loop control variable.

[0042] In the embodiments of the present application, the loop control variable is set, including the current target node identifier (initial value is the root node identifier of the target form control), the current operation type (initial value is the click operation), the loop counter (initial value is zero), the maximum loop count limit (set value is 15-25 times), and the operation history record container. The operation type includes single click, double click, and fill-in.

[0043] In the embodiments of the present application, a multi-round loop interaction mechanism based on feedback is designed. After each operation, the page change situation is obtained and intelligently analyzed, and the next operation strategy is determined according to the analysis result, which can handle complex form controls that require multi-step interaction.

[0044] In step 104, the change information of the page is listened to and collected.

[0045] In the embodiments of the present application, the MutationObserver listener is started to listen to the HTML structure changes of the entire page. The listening configuration parameters include: the childList parameter is set to true to listen to the addition and deletion of child nodes, the subtree parameter is set to true to listen to all descendant nodes, the attributes parameter is set to true to listen to attribute changes, and the attributeOldValue parameter is set to true to record the values before the attribute changes. In the embodiments of the present application, the current target node identifier is located to a specific DOM node, and the current specified operation type is executed, including single click operation, double click operation, text input operation, selection operation, focus operation, etc. The operation execution is realized through the interface of a browser automation tool. The operation execution is realized by a browser automation tool, including at least one of Puppeteer, Selenium, and Playwright.

[0046] In the embodiments of the present application, after the operation is completed, a preset time interval (100-500 milliseconds) is waited, and then the MutationObserver monitoring is ended, and all the page HTML change information recorded during the monitoring is collected.

[0047] Step 105: The collected change information is processed to generate structured difference data.

[0048] In the embodiments of the present application, the collected HTML change information of all pages is classified and processed to generate structured difference data, including the identification and structure information of the added node, the identification information of the deleted node, the identification and change content information of the attribute change node, and the state information of the display and hidden state change node; the added node is supplemented with a unique identifier.

[0049] In the embodiments of the present application, based on the generated structured difference data, a JSON format is used to organize and construct the structured data packet sent to the large language model, which contains four main fields of current node structure state, page change difference data, target filling content information, and historical operation record.

[0050] Step 106: Intelligent analysis and decision are made on the structured difference data.

[0051] In the embodiments of the present application, the API interface of the large language model is called for intelligent analysis and decision, the packaged data packet is input, and the decision result returned by the model is obtained, which includes filling completion state judgment, next step target node identifier, next step operation type suggestion, and decision confidence score.

[0052] Step 107: Filling state judgment.

[0053] In the embodiments of the present application, according to the filling completion state judgment returned by the large language model, if the filling is completed, the loop is exited and a success state is returned; if the filling is not completed, the loop control variable is updated according to the returned suggestion, including the target node identifier and the operation type, and the current operation record is added to the historical record.

[0054] In the embodiments of the present application, it is checked whether the loop counter exceeds the maximum loop number limit, if it exceeds, the loop is exited and a failure state is returned, if it does not exceed, the loop counter is incremented and step 103 is returned to continue execution.

[0055] Figure 2 In order to identify and process the nodes according to the node structure data flowchart of the embodiments of the present application, the following will be described in detail with reference to Figure 2 , the workflow of identifying and processing the nodes to generate the node structure data according to the embodiments of the present application.

[0056] In step 201, the target form control DOM node is acquired.

[0057] In step 202, the current node ID attribute is checked.

[0058] In step 203, according to the current node ID attribute, it is judged whether the node has an ID.

[0059] In the embodiment of the application, if the current node has an ID, the existing identifier is retained and the process proceeds to step 207; otherwise, the process proceeds to the next step.

[0060] In steps 204-206, it is checked whether the HTML node has an ID attribute. If not, a unique ID attribute is automatically generated. The generation rule is a prefix identifier plus a timestamp plus a random number sequence. Depth-first traversal is performed on all descendant nodes to ensure that each node has a unique identifier.

[0061] In step 207, the child nodes are recursively processed.

[0062] In step 208, it is judged whether there are still unprocessed nodes. If yes, the process proceeds to the next step; otherwise, the process returns to step 202.

[0063] In step 209, the simplified data is generated.

[0064] In step 210, the node key attributes are extracted.

[0065] In step 211, non-core information is filtered. In step 212, the hierarchical key mapping relationship between the form fields and the filled content is established.

[0066] In step 213, the JSON structure data is generated.

[0067] In step 214, the simplified node structure data is generated.

[0068] Figure 3 For the listening mechanism and the difference data processing flow according to the embodiment 1 of the application, the following will be described in detail with reference to Figure 3 , the listening mechanism and the difference data processing flow of the embodiment of the application.

[0069] In step 301, the MutationObserver parameter is configured.

[0070] In step 302, the listening setting is performed, including: setting the childList listening to true, setting the subtree listening to true, setting the attributes listening to true, and setting the attributeOldValue to true.

[0071] In step 303, the DOM change is started to be observed.

[0072] In step 304, the form control operation is performed, and a preset time interval is waited In step 305, the listening is stopped, and the obtained DOM change record is processed.

[0073] In step 306, the identification of the newly added node is supplemented.

[0074] In step 307, the structured difference data is generated.

[0075] In step 308, the output is given to the large language model analysis.

[0076] Figure 4 For the intelligent analysis and decision-making process of structured difference data according to the embodiment 1 of the present application, the following will be described with reference to Figure 4 , the intelligent analysis and decision-making process of structured difference data according to the embodiment of the present application will be described in detail.

[0077] In steps 401-402, the large language model input data is constructed and encapsulated.

[0078] In the embodiment of the present application, the generated structured difference data is organized in JSON format, and a structured data package of the large language model is constructed, wherein the structured data package of the large language model includes four main fields of current node structure state, page change difference data, target filling content information, and historical operation record.

[0079] In steps 403-404, the large language model API interface is called, intelligent analysis and reasoning are performed, and a decision result is generated.

[0080] In the embodiment of the present application, the decision result includes filling completion state judgment, next step target node identification, next step operation type suggestion, and decision confidence score information.

[0081] In steps 405-406, the decision result is parsed, and the filling completion state is judged.

[0082] In the embodiment of the present application, according to the filling completion state judgment returned by the large language model, if the filling has been completed, the loop is exited and a success state is returned; if the filling has not been completed, the step 408 is jumped to.

[0083] In step 407, the model suggested operation is adopted.

[0084] In the embodiment of the present application, after the model suggested operation is performed, the step 409 is jumped to.

[0085] In step 408, the conservative default strategy is adopted.

[0086] In step 409, the loop control variable is updated.

[0087] In the embodiment of the application, the loop control variable is updated, including the target node identifier and the operation type, and the current operation record is added to the history record. In step 410, the decision result is returned.

[0088] Figure 5 For the form control listening and filling process flow according to the embodiment 1 of the application, the following will be described with reference to Figure 5 The form control listening and filling process flow of the embodiment of the application is described in detail.

[0089] In step 501, the cascade selector control is triggered to listen to the display of the first-level option panel.

[0090] In step 502, the first-level option list is analyzed.

[0091] In step 503, it is judged whether the target first-level option is found. If yes, go to step 505, otherwise, go to the next step.

[0092] In step 504, more options are searched by scrolling, and the option list is reanalyzed to return to step 503.

[0093] In step 505, the display of the second-level option panel is listened to.

[0094] In step 506, the second-level option list is analyzed.

[0095] In step 507, it is judged whether the target second-level option is found. If yes, go to the next step, otherwise, return to step 502 to analyze the first-level option list.

[0096] In step 508, the display of the third-level option panel is listened to.

[0097] In step 509, the third-level option list is analyzed.

[0098] In step 510, the control filling result is verified Figure 6 For the structure schematic diagram of the web form control intelligent filling system based on page change listening according to the embodiment 1 of the application, as shown in Figure 6 The web form control intelligent filling system based on page change listening according to the embodiment of the application includes an automatic execution layer 601, a browser running environment unit 602, a core technology processing unit 603, and an intelligent decision analysis layer 604, wherein The automatic execution layer 601 includes a browser automation tool, a DOM operation execution interface, an event simulation trigger, and a state verification checker.

[0099] Browser automation tools for programmatically controlling browser behavior, such as Puppeteer, Selenium, Playwright, etc.

[0100] DOM operation execution interface for receiving DOM operation instructions and performing corresponding operations.

[0101] Event simulation trigger to generate event objects based on DOM operation instructions sent by the DOM operation execution interface.

[0102] State verification checker to perform state verification based on event objects dispatched by the event simulation trigger. Through page change monitoring and multi-round verification mechanisms, it can detect the execution results of each operation in real time, discover and handle abnormal situations in a timely manner, and ensure the accuracy and completeness of the fillings.

[0103] Browser runtime environment unit 602 to monitor page changes and generate page change data streams to send to the core technology processing unit 603, including: target web page, DOM tree structure, MutationObserver listener, and page change data stream generation module.

[0104] MutationObserver listener for monitoring HTML structure changes in the entire page, with the following configuration parameters: childList parameter set to true to monitor child node addition and deletion, subtree parameter set to true to monitor all descendant nodes, attributes parameter set to true to monitor attribute changes, and attributeOldValue parameter set to true to record the values before attribute changes.

[0105] Page change data stream generation module to generate page change data streams from page HTML structure changes, including: child node addition and deletion, all descendant nodes, node attribute changes, and recording attribute values before changes.

[0106] Core technology processing unit 603 for node identification, simplified structure data generation, difference detection processing, loop control management, and operation execution, including: node identification module, simplified structure data generation module, difference detection processing module, loop control management module, and operation execution module.

[0107] Node identification module to take the HTML node of the form control as the root node, traverse all descendant nodes, and add unique identifiers to the nodes.

[0108] In the embodiment of the present application, the node identification module checks whether the HTML node has an ID attribute. If not, a unique ID attribute is automatically generated, and the generation rule is prefix identification plus timestamp plus random number sequence. Depth-first traversal is performed on all descendant nodes to ensure that each node has a unique identifier.

[0109] The simplified structure data generation module establishes the hierarchical relationship mapping between nodes, records the parent-child relationship and sibling relationship, and facilitates understanding of the overall structure of the form. All extracted information is organized into JSON format simplified node structure data In the embodiment of the present application, the simplified structure data generation module extracts the key technical information of each node based on the traversal results of the node identification module to generate simplified node structure data for the form control. The key technical information includes node label type, unique identifier, important attribute value, hierarchical relationship position, current state information, etc. Non-core information such as style attributes and event attributes is filtered.

[0110] In the embodiment of the present application, the simplified structure data generation module compresses the complete DOM structure data into lightweight JSON format data to generate simplified structure data.

[0111] The difference detection processing module classifies and processes the collected change information to generate structured difference data. The difference data includes the complete information of the added node, the identifier information of the deleted node, the change details of the attribute change node, and the state information of the display and hide state change node.

[0112] The loop control management module is used for setting the loop control variable.

[0113] In the embodiment of the present application, the loop control variable is set, including the current target node identifier (the initial value is the root node identifier of the target form control), the current operation type (the initial value is the click operation), the loop counter (the initial value is zero), the maximum loop count limit (the set value is 15-25 times), and the operation history record container. The operation type includes single click, double click, and fill in.

[0114] In the embodiment of the present application, a multi-round loop interaction mechanism based on feedback is designed. After each operation, the page change is obtained and intelligently analyzed, and the next operation strategy is determined according to the analysis result, which can handle complex form controls that require multi-step interaction.

[0115] The intelligent decision analysis layer 604 provides decision analysis for filling in the web form control, including: large language model API interface, decision result parser, operation strategy generator, and confidence evaluator.

[0116] The large language model API interface is configured to accept the loop control information from the loop control management module, perform intelligent analysis and decision-making, input the packaged data packet, and obtain the decision result returned by the model. The decision result includes information such as a fill-in completion state judgment, a next step target node identifier, a next step operation type suggestion, and a decision confidence score.

[0117] The decision result parser is configured to parse the decision result from the large language model API interface to obtain information such as a fill-in completion state judgment, a next step target node identifier, a next step operation type suggestion, and a decision confidence score.

[0118] The operation strategy generator is configured to automatically generate an operation strategy based on the information such as a fill-in completion state judgment, a next step target node identifier, a next step operation type suggestion, and a decision confidence score obtained by the decision result parser.

[0119] The confidence evaluator is configured to evaluate the accuracy, reliability, and credibility of the filling result of the web form control. Embodiment 2 This embodiment takes a typical user registration form as an example to illustrate the web form control intelligent filling method based on page change monitoring according to the present application.

[0120] Figure 7 The flowchart of the web form control intelligent filling method based on page change monitoring according to Embodiment 2 of the present application is described below with reference to Figure 7 The web form control intelligent filling method based on page change monitoring according to the present application is described in detail.

[0121] Implementation environment configuration requirements: Operating system: Windows 10 Professional or Ubuntu 18.04 Long Term Support (LTS) or later; Browser environment: Chrome version 90 or later, Firefox version 88 or later, or Edge version 90 or later; Automation tool: Puppeteer version 19.0 or later or Selenium version 4.0 or later; Large language model: GPT-3.5-turbo or a language model with equivalent text understanding capability; Running environment: Node.js version 16 or later or Python version 3.8 or later.

[0122] Target Form Structure Description: The user registration form handled in this example contains the following main fields: username input box, email input box, password input box, confirm password input box, mobile number input box, verification code input box, agree checkbox, submit button, etc. The form is implemented using standard HTML structure and has basic client-side validation functions.

[0123] 1.1 Specific Implementation of Target Determination Step: First, locate the target form control through the DOM query mechanism. In the browser environment, use the document.querySelector method to locate the target form element according to the form's ID attribute or CSS class name. Assuming the HTML identifier of the target form is "registration-form", obtain the DOM node reference of the form through this identifier.

[0124] Next, establish the correspondence between the filled content and the target control. Prepare the data content to be filled in, including: username "testuser123", email "test@example.com", password "SecurePass123", confirm password "SecurePass123", mobile number "13812345678". Organize these data into structured JSON format, establishing a mapping relationship between field names and filled content.

[0125] 1.2 Specific Implementation of Node Identification Step: Use a depth-first traversal algorithm to process all descendant nodes of the target form. The traversal algorithm starts from the form root node and visits each child node in turn. For each visited node, first check whether an ID attribute already exists.

[0126] If the node already has an ID attribute, keep the attribute value unchanged and record it in the node identification mapping table. If the node does not have an ID attribute, generate a unique identifier according to the following rules: use "auto-id" as the prefix, add the current timestamp (millisecond level), and add six random digits to ensure that the generated identifier is unique within the page scope.

[0127] Generated identifier example: "auto-id-1698765432123-892356". Add the generated identifier as the ID attribute to the corresponding DOM node, and maintain an internal mapping table of node identification and DOM node reference to facilitate subsequent quick positioning.

[0128] Count the total number of processed nodes during traversal and record the completion of the identification process. For a typical registration form, usually 15-30 DOM nodes need to be processed.

[0129] 1.3 Implementation of Data Structure Generation Step: Based on the results of the tokenization process, extract key technical information for each node. For each node, extract the following core attributes: node tag name (e.g., input, select, button, etc.), unique identifier (ID attribute value), node type attribute (e.g., text, password, email, etc.), CSS class name information, placeholder text, whether it is required, and other state information.

[0130] Filter out attribute information that is not important for the form filling task, including: style-related attributes (style, color, font, etc.), event handling attributes (onclick, onchange, etc.), and layout-related attributes (position, margin, etc.). Through this selective extraction, the original information of each node is compressed from an average of 200-300 bytes to 50-80 bytes.

[0131] Establish a hierarchical relationship mapping between nodes to record parent-child relationships and sibling relationships, facilitating understanding of the overall structure of the form. Organize all extracted information into a simplified node structure data in JSON format, reducing the overall data volume by more than 70% compared to the complete DOM structure.

[0132] Simplified data structure example: Node information includes an id field to record the unique identifier, a tagName field to record the tag type, a type field to record the input type, a className field to record the CSS class name, a textContent field to record the display text, an attributes field to record key attributes, a children field to record child node information, etc.

[0133] 2.1 Implementation of Initialization Step: Set the initial value of the loop control variable. Set the current target node identifier to the ID value of the first input box that needs to be filled (usually the username input box). Set the current operation type to "click", indicating that the input box needs to be clicked first to obtain focus.

[0134] Initialize the loop counter to 0 and set the maximum number of loop iterations to 20 (considering the moderate complexity of the basic form). Create an empty operation history record array to record the operation content and results at each step.

[0135] Set the current fill-in field identifier to "username" to establish a correspondence between the field name and the fill-in content.

[0136] 2.2 Implementation of Listening Start Step: Create a MutationObserver instance and configure the listening parameters to capture all relevant DOM changes. Set the childList parameter to true to listen for the addition and removal of child nodes; set the subtree parameter to true to listen for changes in all descendant nodes; set the attributes parameter to true to listen for changes in attribute values; and set the attributeOldValue parameter to true to record the original values before attribute changes.

[0137] Set the listening range to the entire document.body to ensure that changes at any location on the page can be captured. Create a change record array to store all the change information collected during the listening period.

[0138] Start the listener to work, at this time the system enters the change listening state, ready to capture subsequent changes caused by page operations.

[0139] 2.3 Specific implementation of the operation execution step: According to the current target node identifier, locate the specific DOM element through the document.getElementById method. Verify whether the target element exists and is operable, if the element does not exist or is disabled, record error information and try the next optional target.

[0140] Perform the corresponding operation according to the current operation type. For "click" operations, call the click method of the element to simulate a mouse click event; for "input" operations, first trigger the focus event to make the element get focus, then set the value attribute of the element and trigger the input event; for "select" operations, set the selected state according to the option value.

[0141] Operation execution is implemented through the API interface of the browser automation tool, ensuring the accuracy and reliability of the operation. Record the execution time and target element information of each operation to facilitate problem diagnosis and performance analysis.

[0142] 2.4 Specific implementation of the change collection step: After the operation is executed, wait for a pre-set time interval of 300 milliseconds, which is enough for the page to complete the relevant change updates, including the display of verification prompts, the linkage changes of other field states, etc.

[0143] Call the disconnect method of MutationObserver to stop listening, and get all the change records collected from the listener. Each change record contains the change type (childList, attributes, etc.), target node reference, and specific content of the change.

[0144] Statistical collection of the total number of change records, for the basic form operation, usually each operation will produce 2-8 change records. If the number of change records is abnormal (too much or too little), it may indicate that the operation execution has a problem and needs to be handled abnormally.

[0145] 2.5 Implementation of difference processing steps: Classify the collected change records. Further divide the childList type of change into two categories: new nodes and deleted nodes, and record the identification information and basic attributes of the related nodes.

[0146] For the newly added node, if the node does not have a unique identifier, generate and add the ID attribute automatically according to the previous rules. Extract the key technical information of the new node, including label type, attribute value, text content, etc.

[0147] Record the attributes type of change as attribute change, including the attribute name, the value before the change, and the value after the change, etc. This kind of change usually reflects the change of form validation state or the change of field availability.

[0148] Check the display and hidden state changes of elements, and judge whether the element changes from display to hidden or from hidden to display by calculating the style or visibility attribute. This kind of change is very important for processing dynamically displayed form fields.

[0149] Organize the results of all classification processing into structured difference data, and encode it in JSON format for subsequent data transmission and analysis.

[0150] 2.6 Implementation of data packaging steps: Build a complete data package sent to the large language model. The data package uses JSON format and contains four main fields: The first field "currentStructure" contains the current node structure state, that is, the simplified form structure data, reflecting the overall situation of the current form.

[0151] The second field "changes" contains the page change difference data, that is, all page changes caused by the operation, providing feedback information of the operation result for the model.

[0152] The third field "targetData" contains the target fill-in content information, that is, the specific data that needs to be filled into the form, helping the model understand the fill-in target.

[0153] The fourth field "history" contains the history operation record, that is, the operation sequence executed before, providing context information for the model.

[0154] Perform size check on the data packet to ensure the data volume is within a reasonable range (usually controlled below 10KB) to avoid transmission delay caused by excessive data size.

[0155] 2.7 Implementation of the intelligent decision-making step: Call the API interface of the large language model and send the encapsulated data packet as input to the model. Set appropriate parameters in the API call, including model version, maximum output length, temperature parameter, etc., to ensure stable and accurate analysis results.

[0156] The model performs intelligent analysis and reasoning based on the input data, understands the current form filling status, analyzes the execution result of the last operation, and determines whether to continue filling and what operation should be performed next.

[0157] Wait for the model to return the decision result, usually within 1-3 seconds. Parse the returned JSON format result and extract four core information: filling completion status judgment (Boolean value), next step target node identifier (string), next step operation type suggestion (string), and decision confidence score (0-1 numerical value).

[0158] Verify the effectiveness of the returned result, check if the necessary fields exist, the numerical value is within a reasonable range, and the target node identifier is valid, etc.

[0159] 2.8 Implementation of the state judgment step: According to the filling completion status judgment returned by the large language model, branch processing is performed. If the returned result indicates that the filling is completed (Boolean value is true), perform form submission verification to check if all mandatory fields have been correctly filled and verify if the data format meets the requirements.

[0160] If the verification is passed, exit the loop and return the filling success status, and record the time of successful completion and the total number of operation steps.

[0161] If the filling is not completed (Boolean value is false), update the loop control variable according to the model's suggestion. Assign the next step target node identifier to the current target node identifier variable and assign the next step operation type to the current operation type variable.

[0162] Add the detailed information of the current operation to the history record, including operation sequence number, target node, operation type, execution time, and change quantity.

[0163] 2.9 Implementation of the loop control step: The loop counter is incremented by 1, and then it is checked whether the maximum number of loop limit is exceeded. If the loop counter value is greater than or equal to the maximum limit value (20 times), it is determined that the filling timeout fails, the loop is exited, and the failure state is returned.

[0164] When the failure state is returned, detailed failure reasons are recorded, including the operation steps that have been performed, the results of the last operation, the remaining fields that have not been filled, and the like, to facilitate problem analysis and debugging.

[0165] If the number of loops is not limited, the step S102.2 is returned to continue to execute the next round of loop, and a new round of page change monitoring and operation execution is started.

[0166] Implementation effect verification: In a standard test environment, the experimental results of using the method of the embodiment to process a user registration form are as follows: Form field recognition accuracy: 98%; single filling success rate: 94%; average processing time: 6.8 seconds; average number of loops: 4-6 times; API call cost: about 0.02 yuan per time.

[0167] Embodiment 3 This embodiment is a detailed description of the cascading selector of the Element UI framework, and shows a preferred technical solution for processing a complex form control that needs multiple-step interaction.

[0168] Figure 8 The flowchart of the web form control intelligent filling method based on page change monitoring according to Embodiment 3 of the present application is as follows: Figure 8 The web form control intelligent filling method based on page change monitoring of the present application is described in detail.

[0169] Application scenario description: process a three-level region selector control, the user needs to select province, city, and county three levels in turn, and the selection of each level will affect the selectable items of the next level. The control is implemented using the el-cascader component of Element UI, and has complex characteristics such as dynamic loading, asynchronous updating, and multi-level expansion.

[0170] Target selection content: select "Guangdong Province" for province, "Shenzhen City" for city, and "Nanshan District" for county.

[0171] Preferred technical parameter configuration: Compared with the basic embodiment, the preferred embodiment makes the following optimization and adjustment in the technical parameters: MutationObserver listens to the interval adjustment of 200 milliseconds, which is shorter than the 300 milliseconds of the basic embodiment, to improve the response speed to rapid changes. This is because complex UI components often involve multiple stages of page updates, and a shorter listening interval can more accurately capture changes in each stage.

[0172] The maximum number of cycles is limited to 25, considering that complex controls require more interaction steps, giving more operation space.

[0173] The confidence assessment threshold is set to 0.85, which is more stringent than the 0.8 of the basic embodiment, improving the accuracy of decision-making requirements and reducing the possibility of misjudgment.

[0174] The difference data sampling frequency is set to 50 milliseconds per time, achieving more refined change detection and ensuring that key state changes are not missed.

[0175] Intelligent node classification identification mechanism: In the node identification processing stage, an intelligent classification mechanism is introduced. According to the CSS class name and HTML structure characteristics of the node, the function type of the node is automatically identified, and the type prefix is added when generating the unique identifier.

[0176] For the cascading selector component, the node is divided into the following types: cascader type represents the main control container, panel type represents the option panel, option type represents the specific option, input type represents the input box, button type represents the trigger button, and container type represents the ordinary container.

[0177] The identifier generation rule is adjusted to: type prefix plus timestamp plus serial number, for example, "cascader-1698765432123-1", "option-1698765432123-15", etc. This classification identification method helps large language models better understand the functional roles of different nodes.

[0178] At the same time, add the data attribute "data-auto-type" to each node to record the function type of the node, which is convenient for subsequent intelligent analysis and operation decision.

[0179] Enhanced difference detection algorithm: Design a special difference detection algorithm for complex UI components that can identify and classify the unique change patterns of cascading selectors.

[0180] Divide the page changes into the following special categories: New option appearance category: When a user selects an option, detect the display of a new option panel or the update of an option list. Record the text content, hierarchical position, and whether it is selectable, etc.

[0181] Option state change category: Detect changes in the activation state, selected state, and disabled state of options. These state changes reflect the progress of user interaction and the response of controls.

[0182] Panel switching category: Detect changes in the display, hiding, and switching of option panels. These changes are crucial for understanding the current selection level.

[0183] Data loading category: Detect the completion of asynchronous data loading, such as the disappearance of loading animations and the update of data lists. Ensure that the next step is taken after the data is fully loaded.

[0184] For each category of changes, establish specialized processing logic to extract relevant technical information and generate more accurate and useful difference data.

[0185] Enhanced large language model prompt strategy: Design a prompt strategy and interaction protocol specifically for cascading selectors to improve the model's ability to understand complex controls.

[0186] In the data sent to the large language model, add the following special fields: The controlType field identifies the control type being processed as "cascader", helping the model understand the control characteristics.

[0187] The targetPath field describes the complete selection path, such as "Guangdong Province > Shenzhen City > Nanshan District", providing a clear target for the model.

[0188] The currentLevel field identifies which selection level is currently at (1 for province level, 2 for city level, 3 for district level).

[0189] The availableOptions field lists all visible options and their attributes, including text content, whether they are selectable, hierarchical information, etc.

[0190] Add special rules to the prompt words: If the target option is seen, click it directly; if the current level of options is incomplete, wait for data loading; if the upper-level option of the target has been selected, check if the lower-level option appears; after selection is complete, confirm the final result, etc.

[0191] Require the model to add a reasoning field in the returned result to explain the reasons and basis for the decision, facilitating the understanding of the model's reasoning process and debugging and optimization.

[0192] Performance improvement effect verification: Through the above optimization measures, the preferred embodiment has achieved significant performance improvement in handling complex controls compared to the basic embodiment: The accuracy of complex control recognition is improved from 88% to 96%, mainly due to intelligent classification identification and enhanced difference detection.

[0193] The success rate of multi-level interaction is improved from 82% to 93%, and the optimized prompt strategy and loop control mechanism play an important role.

[0194] The average processing time is optimized from 12.5 seconds to 8.3 seconds, and more accurate change detection and intelligent decision-making reduce invalid operations.

[0195] The new intelligent rollback recovery mechanism can automatically return to the last stable state and start again when continuous operations fail, and the system failure rate is reduced from 12% to 4%.

[0196] Embodiment 4 This embodiment shows the technical implementation scheme of filling out the form in the complex page environment containing iframe nested structure, focusing on solving the technical problems of cross-frame operation.

[0197] Figure 9 The flowchart of the web form control intelligent filling method based on page change monitoring according to Embodiment 4 of the present application is as follows: Figure 9 The web form control intelligent filling method based on page change monitoring of the present application will be described in detail.

[0198] Application scenario description: processing an order settlement page of an e-commerce website, the main part of the page contains product information and shipping address form, while the payment method selection module is independently nested in an iframe. The user needs to fill in the shipping information in the main page, then select the payment method in the iframe, and finally submit the complete order.

[0199] Technical challenge analysis: the main technical challenges faced by cross-iframe operation include: same-origin policy restrictions prevent direct access to cross-domain iframe content, DOM structures inside and outside the iframe are independent and need to be handled separately, page change monitoring needs to cover multiple frames, and operation results need to be coordinated between frames, etc.

[0200] Cross-iframe node discovery and identification: Implement cross-iframe structure discovery mechanism. First, traverse all iframe elements in the main page, and perform access permission test for each discovered iframe.

[0201] For same-domain iframes (i.e., iframes with the same protocol, domain, and port as the main page), the internal DOM structure can be accessed directly. For these iframes, use the direct manipulation mode: access the internal document object by getting the iframe's contentDocument property, and then perform the same node identification process within the iframe as the main page.

[0202] For cross-domain iframes (iframes that cannot be accessed directly due to the same-origin policy), use the message communication mode: inject a special proxy script into the iframe to establish a two-way communication mechanism based on the postMessage API.

[0203] Assign a unique frame identifier to each iframe, in the format "iframe-" followed by a serial number, such as "iframe-1", "iframe-2", etc. Maintain a frame mapping table that records the identifier, access mode, processing status, and other information for each iframe.

[0204] In the processing of cross-domain iframes, design a special proxy script injection mechanism. The proxy script is responsible for executing node identification, page change listening, operation execution, and other functions within the iframe, and communicates with the main page through the message channel.

[0205] iframe operation execution coordination mechanism: Design a coordination mechanism for cross-frame operation execution. When the target operation node is located within an iframe, the appropriate execution strategy needs to be selected based on the iframe's access mode.

[0206] For same-domain iframes, use the direct DOM manipulation method. Access the internal DOM structure directly through the iframe's contentWindow and contentDocument properties, perform node positioning and operation triggering. This method has the advantages of direct operation and fast response, but has the disadvantage of being limited by the same-origin policy.

[0207] For cross-domain iframes, use the message communication method. The main page sends operation instructions to the iframe through the postMessage API, and the proxy script within the iframe receives the instructions and performs the corresponding operations, then returns the execution results to the main page through the message.

[0208] Design a standardized message communication protocol. The message format contains the following fields: messageId to identify the uniqueness of the message, type to identify the message type (such as EXECUTE_ACTION, REPORT_CHANGES, etc.), targetNodeId to identify the target node, action to identify the operation type, value to pass operation parameters, timestamp to record the message time, etc.

[0209] Implement a reliability guarantee mechanism for message communication. Set a message response timeout (usually 5 seconds), and if no response is received within the timeout, consider the operation a failure. Implement a message retransmission mechanism, and for important operations, automatically retry when failed.

[0210] Technical implementation of iframe internal agent script: Design an agent script that runs inside the cross-domain iframe, which has complete form processing capabilities. The agent script contains the following main functional modules: Message listening module: Listen to operation instruction messages from the main page, parse message content and distribute to the corresponding processing module.

[0211] DOM operation module: Perform basic DOM operations such as node positioning, event triggering, value setting, etc. inside the iframe.

[0212] Change listening module: Start MutationObserver inside the iframe to listen to changes in the iframe.

[0213] Result reporting module: Encapsulate operation execution results and page change information into messages and send them to the main page.

[0214] The agent script ensures coordinated work with the main page in the following ways: After the iframe page is loaded, it actively reports its existence and structure information to the main page; Regularly send heartbeat messages to the main page to confirm normal communication; Confirm the validity of the instruction from the main page before executing critical operations.

[0215] Cross-iframe difference data aggregation: Design an aggregation processing mechanism for cross-frame difference data. Since page changes may occur in the main page and multiple iframes, it is necessary to collect and process change information from different frames uniformly.

[0216] Establish a collection mechanism for multi-frame change data. The MutationObserver in the main page is responsible for listening to changes within the main page range, and the agent script in each iframe is responsible for listening to changes within its own iframe. All change data is finally aggregated on the main page for unified analysis.

[0217] Time synchronization mechanism for design change data. Since there may be a time difference between changes in different frameworks, a unified time reference needs to be established. Use high-precision timestamps to mark each change event, and arrange them in chronological order when aggregating data.

[0218] Implement the impact assessment of change data. Changes in different frameworks have different impacts on the overall filling task, and an evaluation mechanism needs to be established. The main evaluation dimensions include: relevance of changes (whether related to the current filling task), importance of changes (whether it affects subsequent operations), completeness of changes (whether it represents a complete state transition), etc.

[0219] Establish a correlation analysis mechanism for cross-framework changes. Some operations may trigger in one framework, but changes in another framework. For example, selecting a product on the main page may affect the price display in the iframe. The cross-frame correlation needs to be identified.

[0220] Application scenario test results: Under standard test environment, the experimental results of using this deformation embodiment to process cross-iframe form are as follows: The overall success rate of cross-iframe operation reaches 89%, among which the success rate of same-domain iframe operation is 96% and the success rate of cross-domain iframe operation is 82%. The difference in success rate is mainly due to the additional complexity of cross-domain communication.

[0221] The average processing time increases by 2.3 seconds compared to the single-page case, and the increased time mainly comes from the network delay and message processing overhead of cross-domain communication.

[0222] Compatibility test covers Chrome, Firefox, Safari, Edge and other mainstream browsers, with a compatibility of more than 95%.

[0223] Security verification passes special XSS attack protection test, and the message verification mechanism effectively prevents malicious script injection.

[0224] Embodiment 5 For the special interaction characteristics of web applications on mobile devices, the present application provides the following technical adaptation scheme: Touch event processing adaptation: mobile devices mainly interact through touch operations, and need to convert traditional mouse click events into touch events. Implement touch event simulation mechanism, calculate appropriate touch point coordinates according to the position and size of the target element, generate TouchEvent object and distribute it to the target element.

[0225] The coordinate calculation of touch events needs to consider factors such as device pixel ratio, page zoom level, element bounding box, etc., to ensure that the touch point falls accurately within the operable area of the target element.

[0226] Virtual keyboard adaptation processing: When the virtual keyboard of a mobile device pops up, it will change the viewport size and layout of the page, which may cause the target form control to be blocked or moved out of the visible area. Implement a virtual keyboard state detection mechanism to determine the pop-up and folding of the virtual keyboard by listening to window size changes and viewport change events.

[0227] When the virtual keyboard is detected to pop up, automatically adjust the page scroll position to ensure that the currently operated form control remains within the visible area. Use the Element.scrollIntoView method to scroll the target control to the appropriate position.

[0228] Mobile special gesture support: Implement support for mobile-specific interactive gestures such as long press, swipe, zoom, etc. These gestures are used in some mobile form controls to trigger special functions, such as long-pressing to pop up a context menu, swiping to switch options, etc.

[0229] Responsive layout adaptation: Mobile pages usually use responsive design, and the form layout will change under different screen sizes. Implement a responsive layout detection mechanism to dynamically adjust the operation strategy based on the current screen size and device orientation.

[0230] Embodiment 6 For large forms or high-frequency operation scenarios, the following performance optimization strategies are provided: Batch operation optimization mechanism: For complex forms containing a large number of fields, implement batch operation optimization. Organize related operations into batches to reduce the start-stop frequency of MutationObserver and the calling frequency of large language models.

[0231] Design an operation priority queue to handle urgent operations and ordinary operations in a hierarchical manner. Prioritize processing critical steps that affect subsequent operations, and delay processing auxiliary operations that do not affect the main process.

[0232] Cache mechanism optimization: Implement a multi-level cache mechanism to improve processing efficiency. Establish intelligent caching for the response results of large language models, and directly reuse previous decision results when encountering similar form structures and operation scenarios.

[0233] Establish an incremental update mechanism for node structure data to avoid repeated calculation and transmission of known structure information. Only transmit the changed parts to reduce data transmission volume.

[0234] Implement compressed storage of operation history, and use pattern recognition algorithms to identify repeated operation sequences, replacing detailed operation records with simplified patterns.

[0235] Parallel processing optimization: For pages containing multiple independent form regions, implement a parallel processing mechanism. Break down the form into independent sub-tasks, execute them in parallel through multi-threading or multi-processing, and improve overall processing efficiency.

[0236] Establish a task dependency analysis mechanism to identify dependencies between operations and ensure that parallel execution does not violate the logical order of operations.

[0237] Memory optimization strategy: Implement intelligent memory management to release unnecessary DOM references and listener resources in a timely manner. Establish a memory usage monitoring mechanism and actively perform garbage collection when memory usage exceeds the threshold.

[0238] Optimize data structure design and use more compact data representations to reduce memory usage.

[0239] Performance test verification results: Through the above optimization measures, significant performance improvements have been achieved in scenarios involving large forms (more than 50 fields): Processing time has been reduced from 180 seconds before optimization to 45 seconds, with a 75% efficiency improvement.

[0240] Memory usage has been reduced by 40%, with peak memory usage reduced from 120MB to 72MB.

[0241] CPU usage has been reduced by an average of 35%, reducing system resource usage.

[0242] Concurrent processing capacity has been improved by 60%, significantly increasing the number of forms that can be processed simultaneously.

[0243] Experimental data and performance comparison analysis To fully verify the technical effects and advantages of the invention, systematic comparative experiments and performance tests were conducted.

[0244] Test environment standardization configuration: Hardware environment: Intel Core i7-10700K processor (3.8GHz base frequency, 8 cores and 16 threads), 32GB DDR4-3200 memory, 1TB NVMe SSD storage, and gigabit Ethernet connection.

[0245] Software environment: Windows 10 Professional 2004 version, Chrome 110.0.5481.100 browser, Node.js 18.15.0 runtime environment, and Python 3.10.6 development environment.

[0246] Network environment: 100Mbps fiber broadband, stable network connection with a delay of less than 5ms.

[0247] Test website samples: mainstream websites covering 20 different industries, including e-commerce platforms (Taobao, Jingdong, Amazon), online office systems (Dingding, WeChat for Enterprise, Feishu), government service platforms (government websites of various provinces and cities), education platforms (XuetangX, China University MOOC), financial service websites (banks, insurance companies, securities companies), etc., to ensure the representativeness and coverage of the test scenarios.

[0248] Detailed performance test data statistics table:

[0249] The web page form control intelligent filling method and system based on page change monitoring of the present application first systematically applies the MutationObserver page change monitoring technology to the field of automatic form filling, realizing a technical breakthrough from "static recognition" to "dynamic tracking". Traditional methods cannot accurately perceive page changes after executing operations, while the present application can monitor DOM changes caused by each operation in real time, providing accurate feedback information for intelligent decision-making, which is a fundamental technical paradigm shift.

[0250] The web page form control intelligent filling method and system based on page change monitoring of the present application innovatively proposes a simplified node structure data generation algorithm for form filling scenarios, which reduces the data volume by more than 70% through intelligent filtering and key information extraction while ensuring the integrity of technical information. This simplification is not a simple data compression, but an intelligent information filtering based on the characteristics of form filling tasks, which not only solves the efficiency problem of large language models, but also maintains the key information required for decision-making.

[0251] The web page form control intelligent filling method and system based on page change monitoring of the present application designs a cycle interaction mode based on real-time feedback, which processes the multi-step operation process of complex form controls through a multi-round dialogue mechanism. Traditional methods use a "one-time recognition and filling" mode, which cannot handle complex controls that require multi-step interaction, while the present application can gradually complete complex interaction sequences through a cycle mechanism, breaking through the fundamental limitations of traditional modes.

[0252] The web page form control intelligent filling method and system based on page change monitoring of the present application combines the intelligent analysis capability of large language models with the precise execution capability of browser automation technology, forming a closed-loop technical system of "perception-decision-execution-feedback". This combination is not a simple technical superposition, but a special technical architecture designed for the characteristics of form filling tasks.

[0253] Those skilled in the art can understand that the above are only the preferred embodiments of the present application, and are not used to limit the present application, and although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some of the technical features. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for intelligent data entry of web page form controls based on page change monitoring, comprising: Confirm the web form controls and add identifiers to the nodes; Based on the form controls, generate simplified node structure data; Set the loop variable; Use MutationObserver to monitor changes in the HTML structure of the page; Locate the target node based on its identifier and perform the operation. End the listening and retrieve the listening results; Process the monitoring results to generate differential data; Based on the aforementioned difference data and node structure data, intelligent analysis and decision-making are performed; Automatically fill in web form controls based on feedback from the large language model.

2. The intelligent fill-in method for web page form controls based on page change monitoring according to claim 1, characterized in that, The step of adding an identifier to a node in the confirmation webpage form control further includes: Using the HTML node of the form control as the root node, traverse all descendant nodes and add a unique identifier; Check if the HTML node has an ID attribute; if not, automatically generate a unique ID attribute for the node according to the generation rule of prefix identifier plus timestamp plus random number sequence.

3. The intelligent fill-in method for web page form controls based on page change monitoring according to claim 1, characterized in that, The step of adding identifiers to nodes in the confirmation web page form control further includes: for the cascading selector component, the nodes are divided into the following types: cascader type represents the main control container, panel type represents the option panel, option type represents the specific option, input type represents the input box, button type represents the trigger button, and container type represents the ordinary container. Add identifiers to nodes according to the identification rule of type prefix plus timestamp plus sequence number.

4. The intelligent fill-in method for web page form controls based on page change monitoring according to claim 1, characterized in that, The step of generating simplified node structure data based on the form control further includes: traversing all descendant nodes using a depth-first traversal algorithm, extracting key technical information for each node, compressing the complete DOM structure data into lightweight JSON format data, and generating simplified node structure data for the form control.

5. The intelligent fill-in method for web page form controls based on page change monitoring according to claim 4, characterized in that, The key technical information includes: node label type, unique identifier, important attribute values, hierarchical relationship position, current status information, and style attributes and event attributes are filtered.

6. The intelligent fill-in method for web page form controls based on page change monitoring according to claim 1, characterized in that, The step of setting the loop variable further includes: current target node identifier, current operation type, loop counter, maximum loop count limit, and operation history container; the operation type includes: single click, double click, and fill.

7. The intelligent fill-in method for web page form controls based on page change monitoring according to claim 1, characterized in that, The step of using MutationObserver to monitor changes in the HTML structure of the page also includes: Listen to configuration parameters, including: setting the childList parameter to true to listen for the addition and deletion of child nodes; setting the subtree parameter to true to listen for all descendant nodes; setting the attributes parameter to true to listen for attribute changes; and setting the attributeOldValue parameter to true to record the value of the attribute before the change.

8. The intelligent fill-in method for web page form controls based on page change monitoring according to claim 1, characterized in that, The difference data also includes: the identifier and structural information of newly added nodes, the identifier of deleted nodes, the identifier and changed content of nodes with changed attributes, and the identifier of nodes whose display / hidden status has changed.

9. The intelligent fill-in method for web page form controls based on page change monitoring according to claim 1, characterized in that, The step of performing intelligent analysis and decision-making based on the difference data and node structure data further includes: Construct a structured data package for a large language model; the structured data package for the large language model includes: the current node structure state, page change difference data, target input content information, and historical operation records; Call the API interface of the large language model for intelligent analysis and decision-making; Obtain the decision results returned by the model; the decision results include the completion status judgment, the next target node identifier, the next operation type suggestion, and the decision confidence score.

10. The intelligent fill-in method for web page form controls based on page change monitoring according to claim 1, characterized in that, Also includes: For iframes within the same domain, the internal DOM structure can be accessed directly through the iframe's contentWindow and contentDocument properties to perform node positioning and operation triggering; For cross-domain iframes, the main page sends operation instructions to the iframe via the postMessage API. The proxy script inside the iframe receives the instructions, executes the corresponding operations, and then returns the execution results to the main page via a message.

11. The intelligent fill-in method for web page form controls based on page change monitoring according to claim 1, characterized in that, Also includes: The MutationObserver on the main page listens for changes within the main page scope, while the proxy script within each iframe listens for changes within its own iframe.

12. A web form control intelligent fill-in system based on page change monitoring, characterized in that, The intelligent filling method for web form controls based on page change monitoring, as described in any one of claims 1-11, is used to realize intelligent filling of web form controls.

Images