Webpage form intelligent filling method and system based on multi-modal large language model

Abstract

The present application relates to the field of electric digital data processing, and discloses a webpage form intelligent filling method and system based on a multi-modal large language model, comprising: injecting an interactive assistant to analyze a document object model and establishing a mapping relationship between a form filling box and field attributes; calling a multi-modal large language model to analyze input signals to extract a structured information entity set; converting the field attributes and the entity set into feature boundary vectors and high-dimensional semantic vectors respectively; constructing an alignment probability matrix through feature calculation and conditional mask filtering to determine a filling result and write into the filling box, wherein the present application prepositions webpage rigid rule interception and interweaves it into the feature mapping stage, eliminates error filling caused by semantic illusion, solves calculation reflux caused by the misplacement of probability semantics and rigid rules, controls abnormalities and blocks calculation loops through asymmetric verification, and improves the robustness of form data filling.

Description

Technical Field

[0001] This invention relates to a method and system for intelligent filling of web page forms based on a multimodal large language model, belonging to the field of electronic digital data processing technology. Background Technology

[0002] Currently, distributing multi-source data to web form interaction nodes is a common technical approach. This involves using semantic models to extract statistical distribution features of the text stream and calculating the semantic similarity between features and form node labels, thereby establishing a mapping link between information entities and specific rigid database fields. This approach has recognized industrial value in improving data flow efficiency. However, when dealing with complex and heterogeneous web forms with high dynamic variation, the front-end page frequently adds or deletes nodes or changes attributes, and node labels exhibit non-standard naming, directly causing static mapping templates to fail. Simultaneously, the probabilistic tendency of language models when decoding unstructured signals can easily generate semantic illusion data that does not conform to the rigid boundaries of the target page during the feature generation stage. When these digital signal streams are injected into the front-end interactive interface, they cannot be intercepted by hard physical rules such as character length control or data type validation, leading to frequent error alarms and logical rollbacks, potentially causing the system to fall into the trap of repeated parsing and computational deadlock.

[0003] To address these bottlenecks, common approaches focus on manually expanding the keyword dictionary or linearly increasing the language model parameter size to improve the recognition generalization boundary. However, this not only exhibits lag but also brings drawbacks such as high deployment costs and increased response latency. Furthermore, it fails to eliminate the substantial misalignment between the probability output and the rigid front-end validation rule space. This approach, which avoids architectural flaws, fails to address the core issue. While the aforementioned architecture can reduce some manual data entry workload under normal operating conditions, it is not only limited by the hardware limitations of the underlying support structure but also has shortcomings at the software level. For example, the Chinese patent with publication number CN116992081A... The invention patent application discloses a data processing method, device, and user terminal for a web page form. The above solution implicitly relies on the static and constant preset of the web page topology structure. It uses a large cloud model to perform probabilistic semantic extraction of material data to fill the form. However, in actual high-dynamic heterogeneous web page interaction scenarios, the front-end page frequently experiences transient node mutations and asynchronous attribute refreshes. This control architecture, which separates probabilistic space decoding from the rigid front-end rules, is prone to outputting signal flows that violate the physical boundaries of the form due to semantic illusions, causing frequent verification rollbacks and computing power deadlocks. It cannot meet the objective reality of the need for robust topology alignment under complex working conditions.

[0004] Therefore, the technical problem to be solved by this invention is how to establish a data flow self-verification interactive closed loop by weaving front-end hard rules into the probabilistic feature decoding stage. Summary of the Invention

[0005] To address the problems in the background art, the technical solution of this invention is as follows: A method for intelligent filling of web page forms based on a multimodal large language model, comprising the following steps: Step S1: Inject a floating interactive assistant into the target webpage through a browser plugin to serve as a floating interactive entry point. Then, parse the document object model structure data of the target webpage, establish a mapping relationship between form fill boxes and field attributes, and capture node changes in the document object model structure data of the target webpage through the document object model change listening interface to update the mapping relationship. Step S2: Receive multimodal input signals, call the fusion multimodal interaction module to perform cross-modal feature normalization and temporal alignment on the multimodal input signals, and call the multimodal large language model to perform joint visual and semantic parsing on the multimodal input signals to extract structured information entity sets; Step S3: Extract the data type, length limit, required attributes, regular expression rules, enumeration value set, numerical range, field dependency relationship, and cross-field logic rules of the form fill box, and perform hierarchical mapping to classify the field data type, length limit, and required attributes into the first level of the basic level, the regular expression rules, enumeration value set, and numerical range into the second level of the complex level, and the field dependency relationship and cross-field logic rules into the third level of the field level. Then, construct multi-level deterministic feature boundary vectors in a hierarchical manner, and transform the structured information entity set into a high-dimensional semantic feature vector. Step S4: Obtain the semantic vector of the form fill box. The semantic vector is obtained by extracting the semantic connotation, structural attributes, rule constraints and topological context information of the form fill box and fusing them. Then, it is mapped to a unified high-dimensional numerical space using an embedding encoding technique that is the same as that of the multimodal large language model. Calculate the dot product between the high-dimensional semantic feature vector and the form fill box semantic vector to generate an intermediate semantic association matrix. Apply a conditional masking operator to the intermediate semantic association matrix based on the multi-level deterministic feature boundary vectors to perform masking filtering and generate an alignment probability matrix. Format the pre-filled result of the entire form according to the alignment probability matrix. Review the pre-filled result to determine the final filling result and input the final filling result into the corresponding form fill box.

[0006] Preferably, the process of determining the final filling result based on the alignment probability matrix in step S4 also includes a feedback convergence stage: Step S41, when the alignment probability corresponding to the alignment probability matrix is ​​lower than the preset alignment probability threshold and the attribute of the corresponding structured information entity satisfies the required attribute, the local topological feature components of the hierarchical belonging node and the context adjacent node of the form filling box in the current document object model tree structure are retrieved, and the local topological feature components are directly superimposed on the high-dimensional semantic feature vector of the structured information entity.

[0007] Preferably, the process of parsing the document object model structure data and updating the mapping relationship in step S1 specifically includes: parsing the document object model structure data of the target webpage, capturing the addition, deletion and attribute modification status of each node in the document object model structure data of the target webpage through the document object model change monitoring interface, and triggering the re-parsing process of the document object model structure data to update the mapping relationship.

[0008] Preferably, step S2 includes the following sub-steps: step S21, receiving a multimodal input signal composed of text, speech, images, and form screenshots; step S22, converting the speech in the multimodal input signal into natural language text, and calling a multimodal large language model to perform joint visual and semantic analysis on the images and form screenshots in the multimodal input signal, extracting the text fields and handwritten content, resolving the omissions and ambiguous expressions in the natural language text, and outputting a structured information entity set.

[0009] Preferably, step S3 includes the following sub-steps: Step S31, discretize the document object model structure data and extract the field data type, length limit, required attributes, regular expression rules, enumeration value set, numerical range, field dependency relationship, and cross-field logic rules corresponding to the form fill box; Step S32, perform hierarchical mapping processing on the extraction results of step S31, so that the field data type, length limit, and required attributes are classified into the first basic level, the regular expression rules, enumeration value set, and numerical range are classified into the second complex level, and the field dependency relationship and cross-field logic rules are classified into the third field level; Step S33, continue to perform vectorization construction processing on the hierarchical mapping results of step S32 to obtain multi-level deterministic feature boundary vectors, and at the same time transform the structured information entity set into a high-dimensional semantic feature vector.

[0010] Preferably, step S4 further includes the following sub-step: step S42, if the text length of the corresponding structured information entity exceeds the corresponding length limit and the data format of the structured information entity does not meet the corresponding field data type, the weight value of the corresponding matching channel in the alignment probability matrix is ​​set to 0.

[0011] Preferably, the method further includes the following steps: Step S5, after the form filling box is pre-filled, a global review process is triggered in the sandbox created by the browser plugin, the corresponding control flow branch is determined through dual-path validation logic, a secondary interactive prompt is output for the form filling box whose alignment probability is lower than the preset alignment probability threshold, and the form page of the target webpage is controlled to complete the data submission based on the received confirmation feedback signal.

[0012] Preferably, step S41 includes the following sub-steps: Step S411, extract the parent node features, child node features, and semantic attributes of the form fill box in the document object model tree, as local topological feature components of the hierarchical belonging node and the contextual adjacent nodes.

[0013] Preferably, after the interactive assistant is deployed in step S1, the following steps are continuously executed throughout the entire runtime of the target webpage: Step S6, during the runtime of the target webpage, the interactive assistant is kept in a floating interactive state and keeps in real-time linkage with the form page. When the target webpage is redirected or asynchronously refreshed, the parsing process of the new form page is triggered and the mapping relationship is updated.

[0014] A web form intelligent filling system based on a multimodal large language model includes a memory and a processor. The memory stores a computer program, and the processor calls and executes the computer program in the memory to implement the web form intelligent filling method based on a multimodal large language model.

[0015] Compared with the prior art, the beneficial effects of the present invention are: 1. In intelligent form filling based on a multimodal large language model, a real-time data capture mechanism is constructed through the collaborative cooperation of the document node parsing interface and the mutation monitoring interface. Under dynamic conditions where nodes are added, deleted, or attributes are changed on the target page, the mutation monitoring interface adaptively senses structural changes and triggers a re-parsing process in situ, extracting all field names, data types, and validation topologies. It automatically establishes a monotonic mapping relationship between the form body and field attributes, eliminating the need for manual configuration of fixed templates. This enables adaptive topology structure for heterogeneous dynamic forms, avoiding data parsing interruptions or fill alignment failures caused by sudden changes in form structure, and improving the topological robustness of complex heterogeneous data processing systems in dynamic interconnected network environments.

[0016] 2. By integrating the visual encoding capabilities of the multimodal interaction module with the deep semantic extraction mechanism of the natural language processing unit, a collaborative heterogeneous information integration path is opened up. The multimodal large language model receives heterogeneous signals such as handwritten photos, text input, voice input, and page screenshots, converts the pixel array in the visual image into a character sequence carrying multidimensional semantic features, and performs joint decoding with the context text to resolve ellipsis and ambiguous expressions in natural language, outputting a high-density structured entity dataset. This overcomes the limitation of traditional automatic data filling methods that can only deal with single-modal standard text input, completes the steady-state convergence of cross-modal heterogeneous signals to discrete feature streams, and improves the accuracy of low-level information extraction in non-standard open interaction scenarios.

[0017] 3. In the semantic vector similarity matching process, a logical conditional masking operator is introduced to construct an adaptive directed conditional mapping mechanism from a high-dimensional probability space to a low-dimensional deterministic rule boundary. The system combines the language vector of the information entity with the attribute vector of the frame node and calculates the similarity to generate a probability matrix. Based on the character length, data type, and deterministic vector of the enumeration boundary, spatial hard blocking is implemented. When the confidence level is lower than the preset alignment probability threshold, the context component of the cascade node is retrieved to implement directed bias compensation. The front-end rule constraints of the webpage are woven into the feature decoding stage in situ, avoiding the incorrect filling of non-compliant digital streams due to semantic illusion, and eliminating the logical concessions and system overhead caused by the original error interception of the webpage during the data stream submission stage. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the overall steps of the intelligent web form filling method based on a multimodal large language model according to the present invention. Figure 2 This is a logical architecture diagram of the intelligent form filling method for web pages based on a multimodal large language model according to the present invention.

[0019] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0021] A method for intelligent form filling on web pages based on a multimodal large language model, such as... Figures 1-2 As shown, it includes the following steps: Step S1: Inject a floating interactive assistant into the target webpage through a browser plugin to serve as a floating interactive entry point. Then, parse the document object model structure data of the target webpage, establish a mapping relationship between form fill boxes and field attributes, and capture node changes in the document object model structure data of the target webpage through the document object model change listening interface to update the mapping relationship. Step S2: Receive multimodal input signals, call the fusion multimodal interaction module to perform cross-modal feature normalization and temporal alignment on the multimodal input signals, and call the multimodal large language model to perform joint visual and semantic parsing on the multimodal input signals to extract structured information entity sets; Step S3: Extract the field data types, length limits, required attributes, regular expression rules, enumeration value sets, numerical ranges, field dependencies, and cross-field logic rules of the form input boxes, and perform hierarchical mapping to classify the field data types, length limits, and required attributes into the first-level basic level, the regular expression rules, enumeration value sets, and numerical ranges into the second-level complex level, and the field dependencies and cross-field logic rules into the third-level field level. This process constructs multi-level deterministic feature boundary vectors and transforms the structured information entity set into a high-dimensional semantic feature vector. Step S4: Obtain the semantic vector of the form fill box. The semantic vector is obtained by extracting the semantic connotation, structural attributes, rule constraints and topological context information of the form fill box and fusing them. Then, it is mapped to a unified high-dimensional numerical space using an embedding encoding technique that is the same as that of the multimodal large language model. Calculate the dot product between the high-dimensional semantic feature vector and the form fill box semantic vector to generate an intermediate semantic association matrix. Apply a conditional masking operator to the intermediate semantic association matrix based on the multi-level deterministic feature boundary vectors to perform masking filtering and generate an alignment probability matrix. Format the pre-filled result of the entire form according to the alignment probability matrix. Review the pre-filled result to determine the final filling result and input the final filling result into the corresponding form fill box.

[0022] Preferably, the process of determining the final filling result based on the alignment probability matrix in step S4 also includes a feedback convergence stage: Step S41, when the alignment probability corresponding to the alignment probability matrix is ​​lower than the preset alignment probability threshold, and the attribute of the corresponding structured information entity satisfies the required attribute, the local topological feature components of the hierarchical belonging node and the context adjacent node of the form filling box in the current document object model tree structure are retrieved, and the local topological feature components are directly superimposed on the high-dimensional semantic feature vector of the structured information entity.

[0023] Preferably, the process of parsing the document object model structure data and updating the mapping relationship in step S1 specifically includes: parsing the document object model structure data of the target webpage, capturing the addition, deletion and attribute modification status of each node in the document object model structure data of the target webpage through the document object model change monitoring interface, and triggering the re-parsing process of the document object model structure data to update the mapping relationship.

[0024] Preferably, step S2 includes the following sub-steps: step S21, receiving a multimodal input signal composed of text, speech, images, and form screenshots; step S22, converting the speech in the multimodal input signal into natural language text, and calling a multimodal large language model to perform joint visual and semantic analysis on the images and form screenshots in the multimodal input signal, extracting the text fields and handwritten content, resolving the omissions and ambiguous expressions in the natural language text, and outputting a structured information entity set.

[0025] Preferably, step S3 includes the following sub-steps: Step S31, discretize the document object model structure data and extract the field data type, length limit, required attributes, regular expression rules, enumeration value set, numerical range, field dependency relationship, and cross-field logic rules corresponding to the form fill box; Step S32, perform hierarchical mapping processing on the extraction results of step S31, so that the field data type, length limit, and required attributes are classified into the first basic level, the regular expression rules, enumeration value set, and numerical range are classified into the second complex level, and the field dependency relationship and cross-field logic rules are classified into the third field level; Step S33, continue to perform vectorization construction processing on the hierarchical mapping results of step S32 to obtain multi-level deterministic feature boundary vectors, and at the same time transform the structured information entity set into a high-dimensional semantic feature vector.

[0026] Preferably, step S4 further includes the following sub-step: step S42, if the text length of the corresponding structured information entity exceeds the corresponding length limit and the data format of the structured information entity does not meet the corresponding field data type, the weight value of the corresponding matching channel in the alignment probability matrix is ​​set to 0.

[0027] Preferably, the method further includes the following steps: Step S5, after the form filling box is pre-filled, a global review process is triggered in the sandbox created by the browser plugin, the corresponding control flow branch is determined through dual-path validation logic, a secondary interactive prompt is output for the form filling box whose alignment probability is lower than the preset alignment probability threshold, and the form page of the target webpage is controlled to complete the data submission based on the received confirmation feedback signal.

[0028] Preferably, step S41 includes the following sub-steps: Step S411, extract the parent node features, child node features, and semantic attributes of the form fill box in the document object model tree, as local topological feature components of the hierarchical belonging node and the contextual adjacent nodes.

[0029] Preferably, after the interactive assistant is deployed in step S1, the following steps are continuously executed throughout the entire runtime of the target webpage: Step S6, during the runtime of the target webpage, the interactive assistant is kept in a floating interactive state and keeps in real-time linkage with the form page. When the target webpage is redirected or asynchronously refreshed, the parsing process of the new form page is triggered and the mapping relationship is updated.

[0030] A web form intelligent filling system based on a multimodal large language model includes a memory and a processor. The memory stores a computer program, and the processor calls and executes the computer program in the memory to implement the web form intelligent filling method based on a multimodal large language model.

[0031] Example 1: In the scenario of handling government service matters, the intelligent form filling system for web pages deploys a floating interactive assistant directly to the target web page as a floating interactive entry point through a browser plugin during step S1, thereby achieving dynamic injection of the interactive assistant. Subsequently, the system uses the web page document object model parsing interface to identify the full document object model structure data of the current page form in real time, including all fill boxes and their corresponding attributes, including name, contact number, and application material list. The system automatically establishes a mapping relationship for the above fill boxes. This mapping relationship records the node unique identifier, field name, data type, and required attributes of each fill box. If the user is submitting an application during the application process... When a dynamic change to the form page is triggered, such as the addition of a temporary residence address input box related to residence permit processing, the system automatically captures the addition, deletion, and attribute modification status of each node in the target webpage's document object model structure data through the document object model change monitoring interface to accurately track node changes and update the mapping relationship in real time, thereby maintaining real-time adaptation to the dynamic structure form. After completing the deployment in step S1, the system continuously keeps the interactive assistant in a floating interactive state and maintains real-time linkage with the form page throughout the entire runtime of the target webpage. When the target webpage undergoes page redirection or asynchronous refresh, the parsing process of the new form page is triggered and the mapping relationship is updated.

[0032] Users submit long text messages containing their application requirements through an interactive portal. The text includes the application for a residence permit, along with the temporary address and contact number. The system utilizes a built-in multimodal interaction module to receive and normalize the long text message across modal signals. This module consists of a front-end floating interactive component, a multi-source signal cascade path, and a feature pre-allocation unit. It performs high-dimensional data alignment on the input text, speech, images, or screenshots of page forms. Subsequently, the system calls a multimodal large language model to perform joint visual and semantic analysis of the input signals, converting the text into structured information entities. The system executes step S3 for hierarchical deconstruction, which specifically includes the following sequential processing sub-stages: First, in step S31, the document object model structure data is discretized, extracting the field data types, length limits, required attributes, regular expression rules, enumeration value sets, numerical ranges, field dependencies, and cross-field logical rules of the form fill boxes. Then, in step S32, the extracted results are subjected to hierarchical mapping processing, with the field data types, length limits, and required attributes serving as the first-level base layer, and the regular expression rules, enumeration value sets, and numerical ranges serving as the second-level base layer. The complex hierarchy of the second layer, field dependencies, and cross-field logical rules are used as the third layer's field hierarchy for hierarchical construction. Finally, in step S33, vectorization processing is performed to obtain multi-level deterministic feature boundary vectors with a hierarchical topology. Simultaneously, the system transforms the structured information entity set into high-dimensional semantic feature vectors. During this process, to establish the mathematical validity of feature comparison, the system uses an embedding encoding technique derived from the multimodal large language model when obtaining the semantic vector of the form fill box. Its core technology lies in: the embedding used to extract and construct the semantic vector of the form fill box. The input encoding model shares the same word segmenter vocabulary, high-dimensional vector space mapping matrix parameters, and pre-trained weight benchmarks with the underlying text and visual modal embedding layers of the multimodal large language model. That is, it directly calls the forward propagation calculation of the embedding layer of the multimodal large language model itself, thereby forcibly constraining the discrete webpage topology rules, frame context, and structured information entity set extracted by the large model into a completely unified high-dimensional numerical space with the same metric. Then, the system calculates the dot product of the high-dimensional semantic feature vector and the frame semantic vector of each form filling box to generate an intermediate semantic association matrix.

[0033] For the intermediate semantic association matrix, the system applies a conditional masking operator to perform logical filtering based on the multi-level deterministic feature boundary vector, which is composed of the first-level basic layer, the second-level complex layer, and the third-level field layer. For candidate matching channels that violate the first-level basic layer constraints due to exceeding character length limits or missing required attributes, or violate the second-level complex layer constraints due to data format inconsistencies, the system sets the corresponding matrix elements to 0 to achieve spatial hard blocking. This completes the rule interweaving filtering of the underlying physical and compliance layers in the pre-feature mapping stage. Specifically, the in-situ weaving of front-end rule constraints into the feature decoding stage means that after the system receives the original aligned components of the multimodal large language model output without activation functions on the browser plugin side, it does not directly perform hard text post-processing interception. Instead, it uses the multi-level deterministic feature boundary vector, which contains rigid boundaries and integrates the first-level basic layer components, the second-level complex layer components, and the third-level field layer components, as a conditional probability operator to filter field data types, length limits, required attributes, regular expression rules, etc. The constraints of eight core dimensions—enumerated value set, numerical range, field dependency relationship, and cross-field logical rules—are layered and projected onto a unified feature space. A Hadamard product operation at the matrix level is then performed with the original alignment components. This completes the hierarchical projection filtering of rigid rules while generating the probability distribution. Based on this, as a deep feedback convergence stage in the overall alignment probability matrix calculation process of step S4, if the alignment probability of a required field is lower than 0.85, the system retrieves the local topological feature components of the parent node and adjacent nodes of the form field in the document object model tree structure. This feature is then directly superimposed onto the feature vector of the structured information entity, and the matching probability is recalculated until an alignment probability matrix conforming to the rule constraints is generated. This direct superposition refers to vector merging within the same unified alignment high-dimensional feature space. Specifically, since the local topological feature components of the document object model tree belong to discrete structural graph data, while the feature vectors of the structured information entity belong to the multimodal text embedding features of a large model, the system sets a cross-dimensional bridging linear projection matrix before superposition. The projection matrix is ​​pre-stored in the browser plugin and is used to normalize and project the structural graph components onto a 128-dimensional space with the same dimension as the text embedding. This eliminates the differences in dimensions and data structures between the topological structure and the text semantics. After spatial alignment, the corresponding dimension values ​​of the two sets of vectors are algebraically added, ensuring the mathematical and logical consistency of the operation loop under multidimensional constraints. After generating the alignment probability matrix, the system first selects the feature channel with the highest alignment probability for single mapping and binding, thereby formatting and generating the pre-filled result of the full form in memory. The pre-filled result is used as an intermediate review object and input into the subsequent review process. Before data submission, the system starts the global review process. Specifically, the dual-path verification logic in the global review process includes a system exception control branch and a subjective correction response branch. When the form is pre-filled, the system triggers the objective compliance check of the first path, automatically comparing the boundary vectors of each box with the filled content. If it is determined that there is a missing required field or the format verification fails, the system exception control branch is switched to. When the user triggers an active editing event on the interactive interface, the system... The system initiates a second path for subjective control verification, identifying the action as a user-initiated correction and unconditionally redirecting to the subjective correction response branch. These two parallel logical paths collaborate to complete differentiated routing, ensuring the closed-loop operation of the human-machine collaborative interaction control flow. For the anomaly of missing required application materials, the system displays anomaly details and provides correction suggestions to the user. After user confirmation, the system automatically updates the form and resets the verification status. If the user is not satisfied with the pre-filled temporary address, they can enter new address information through the interactive entry. The system directly responds to this modification instruction, updates the corresponding fill box, and proactively stops the automatic verification process for the modified content after completing this modification. The final definitive data after correction or confirmation through the dual-path verification logic is output as the final filling result and formally written and input into the corresponding form fill box through the browser plugin's built-in automation interface. The system records the operation log throughout the filling process, including automatic filling records, user instruction modification records, and anomaly correction feedback. This log is stored in the local browser cache to ensure that the data is only accessed within the current filling session.

[0034] Example 2: When the intelligent form filling system processes large-scale heterogeneous form data streams, the accuracy and logical stability of the processing are verified. An automated testing platform based on a web server is set up, configured with a multi-task concurrent processing engine to simulate the real-time concurrent pressure of multimodal input signals in a real network environment. To simulate an industrial-grade data environment, signal links containing Gaussian white noise and pseudo-random packet loss characteristics are actively injected into the input signal stream to simulate transmission interference present in real business scenarios. The experiment is divided into an experimental group and a control group. Both groups use web forms from the same source as test objects. The control group uses a data filling method based on traditional static document object model template matching, while the experimental group uses the filling method proposed in this invention based on multimodal large language model feature mapping and deterministic feature boundary vector filtering. In the test, the core problem variable, namely the complexity of the form node structure, is set to three gradients: low complexity (10 fields), medium complexity (50 fields), and high complexity (more than 100 fields).

[0035] Experimental data shows that as the complexity of the form node structure increases, the fill accuracy of the control group exhibits a stepwise decline. In high-complexity tests, due to the inability to handle dynamic changes in node attributes in real time, its fill success rate drops below 45.0%, and numerous logical validation rollbacks occur due to errors in document object model structure recognition. Under the same high-complexity conditions, the experimental group successfully maintained a fill accuracy of 97.2% to 97.5% by identifying changes in document object model nodes in real time and dynamically updating the mapping relationship. In the verification of the effectiveness of deterministic feature boundary vectors, two extreme parameter control groups were set up, with the first control group matching... With the confidence threshold set to 0.60, the results showed that 12.3% of field filling logic confusion occurred due to the filter operator's misreception of low-quality matching information. The second control group increased the matching confidence threshold to 0.98. The results showed that the excessively high threshold triggered too frequent adaptive degradation optimization strategies, causing the average processing latency of a single form to surge from the original 2.5s to 8.2s. Data analysis showed that when the matching confidence parameter deviated from the preset range, the system's processing efficiency and data accuracy both decreased sharply, verifying the optimal working window effect of the 0.85 setting value in balancing multi-dimensional feature alignment and processing efficiency.

[0036] In multimodal data input processing, the system receives multimodal input signals composed of text, speech, images, and form screenshots by calling an independently deployed fusion multimodal interaction module. This fusion multimodal interaction module incorporates a visual encoding unit composed of a deep convolutional neural network and a natural language processing unit composed of a bidirectional Transformer architecture. It utilizes the visual encoding capability of the visual encoding unit to convert image pixel arrays into character sequences carrying multidimensional semantic features, and leverages the deep semantic extraction mechanism of the natural language processing unit to perform context-dependent encoding on the translated text and speech content, thereby opening up a cross-modal collaborative heterogeneous information integration path. Experiments show that the high-dimensional semantic feature vector processed by the multimodal large language model can effectively distinguish between semantically similar but mutually exclusive fill boxes when performing a dot product operation with the box semantic vector. When the input signal contains ambiguous referential information present in the image screenshot, the system successfully increased the matching probability from the initial 0.72 to 0.91 by weighted bias compensation of the topological feature components of adjacent nodes. The above results confirm that in an engineering environment where noise interference and complex structural changes coexist, by weaving the rigid rule boundary vector into the feature mapping process in advance, this scheme eliminates erroneous matching in the semantic parsing process and ensures the robustness of the topological alignment of the data flow under multidimensional constraints.

[0037] Example 3: The current intelligent web form filling system is deployed in complex web form processing environments with high real-time requirements. To cope with implicit negative event interference in network transmission, the system constructs a control closed loop including a document object model parsing module, a semantic parsing module, and a logic execution module to achieve robust control over the form input process. The document object model parsing module mainly consists of a webpage DOM traversal unit, a dynamic change tracking listener, and an attribute feature mapping register. Its core function is to parse the document object model tree structure data of the target webpage in real time through a dedicated browser plugin and to capture transient page changes in real time using an advanced event-driven mechanism. The system dynamically establishes and updates a monotonic mapping relationship between form input boxes and field attributes. Specifically, it calls the host browser's underlying DOM interface, runs persistently throughout the page's lifecycle, and stores the parsed attribute data in a structured manner in a local cache. The semantic parsing module mainly consists of a heterogeneous signal preprocessing unit, a deep visual multimodal encoder, and a contextual semantic resolution network. Its core function is to receive and integrate multimodal input signals from the user, completing a deep joint parsing of visual and semantic data. Specifically, it employs a neural network model with a deep attention mechanism to perform feature analysis on the signals at the server or edge computing gateway. Extracting and projecting vector space eliminates ambiguity and outputs a high-density set of structured information entities in steady state. The logic execution module mainly consists of multi-level rule feature operators, conditional mask filtering matrices, and dual-path consistency review units. Its core function is to weave the rigid physical rules of the front-end page into the probabilistic feature decoding process of semantic vector similarity matching. Specifically, it calculates the dot product of the high-dimensional semantic feature vector and the frame semantic vector in the local sandbox environment to generate an intermediate matrix, and applies a conditional mask operator in the form of a Hadamard product to block non-compliant electronic digital signal streams, thereby forming a self-verifying closed-loop control. In the specific operation control, the system... Upon receiving the structured information entity set, the system performs pre-validation on the extracted field data based on the document object model structure of the target form. The system pre-stores a parameter benchmark library containing unique identifiers of each field node, required attributes, and preset input constraints. This benchmark library is constructed through statistical fitting of historical form structures. The system calculates the similarity between the semantic feature vector of the input information entity and the semantic vector of each fill box. Before performing the mapping operation, the system triggers a logical judgment procedure for implicit negative events. The system transforms the preset input constraints of each field into deterministic feature boundary vectors and executes a conditional mask operator to filter out fill paths in the input data that violate the preset input constraints.

[0038] When the system identifies an implicit negative event record in the input information entity containing logical contradictions or format deviations from preset specifications, the system applies a reverse correction strategy. The system calculates the cosine similarity between the feature vector of the input data and a preset negative feature library. If the similarity exceeds a preset interception threshold of 0.75, the system determines the record as a negative event and triggers downgrade processing. This downgrade processing involves the system blocking the automatic data entry path for that data and automatically generating a verification request containing specific field names, missing attributes, and modification suggestions, which is then attached to the corresponding input box on the user interface. In the experimental testing of this logic, a typical e-government application scenario was selected, and application data containing logical contradictions was input, i.e., within the application deadline... However, the enrollment time was not filled in. After the system detected this implicit negative event, it triggered a control logic switch in the processing chain, setting the autofill priority of the field to zero. At the same time, it sent an exception feedback instruction to the interface layer through the interface. This instruction drove the front-end interaction unit to highlight the current fill box in orange and pop up a window prompting the user to check the time dimension logic. After the user corrected the data, the system re-performed the matching and verification of the feature boundary vector and semantic feature vector. Through this pre-intervention of logic, the system achieved robust control over the data source of the web form, ensuring that the content finally filled in the form fully complies with the deterministic requirements of the backend business rules, and eliminating the interruption of calculation backflow and verification after form submission caused by implicit negative events.

[0039] Example 4: Before automating large-scale web form data processing, the system executes an offline data baseline calibration procedure to eliminate background noise in multimodal input signals and establish initial parameters for the processing model. The system extracts 500 typical samples from an existing historical form input sample set, and through statistical analysis, extracts the input response time distribution characteristics of each fill box, establishing an input frequency distribution model covering 10 basic fields such as name, address, and date. The system solidifies the model parameters into a baseline reference table and loads it into the local computing unit during system deployment as the initial reference for subsequent real-time data stream anomaly detection. During the automated filling of web form data, the system establishes a real-time monitoring link for the interaction process of each fill box. Every 20ms, the system traverses the document object model structure of the target form to detect whether the fill box attributes have changed and counts the user's switching frequency between fill boxes. When the system captures that the semantic feature vector of the user input data deviates from the preset feature distribution in the baseline reference table by more than 30%, the system triggers a parameter reconstruction procedure, and the system establishes a first-in-first-out data in memory. The buffer queue serves as a delay buffer medium, with a time buffer window set to 500 to 1000 milliseconds. Within this window, node change data generated from multiple traversals is accumulated. This change data is then merged to eliminate high-frequency noise, converting discrete changes into a periodic, stable signal stream. Specifically, merging change data involves deduplication and temporal state fitting based on a time sliding window. Within the 500 to 1000 millisecond buffer period, the system merges transient jitter or repetitive change events caused by asynchronous loading captured from multiple consecutive document object model tree traversals, retaining only the final attribute state snapshot of the node at the end of the window. This filters out discrete high-frequency transient noise caused by network rendering latency, providing a periodic, stable, and deterministic signal input for subsequent parameter reconstruction. To adjust the decoding probability bias of the multimodal large language model for heterogeneous and variable form environments in real time and quantitatively, the system needs to dynamically establish a feature decoding weight parameter W to control the linear scaling of the attention mechanism layer's projection matrix during parameter reconstruction. The parameter reconstruction procedure adjusts the decoding matrix based on the weighted linear superposition principle. The calculation formula is as follows: ,in, This is the feature decoding weight parameter for the large language model, with a value range of 0 to 1; This represents the field density of the current page's form, and its value range is an integer greater than 0. The degree of contextual relevance of the fill box is determined by the topological distance between neighboring nodes in the document object model tree, with a value ranging from 0 to 1; and These are the field density adjustment factor and the context association adjustment factor, respectively, and they satisfy the constraint that their sum equals 1. The weight parameters are calculated according to the formula above. The system updates the projection matrix in the attention mechanism layer of the model, changes the feature decoding distribution bias, and calculates and updates the feature decoding weight parameters of the multimodal large language model in real time based on the field density of the current page form and the contextual relevance of the fill box.

[0040] To ensure system stability, the system employs cross-validation rules based on field completeness and logical relevance for each web form's processing logic. The system compares input information from related fields on the same page to verify for logical inconsistencies between input items. In the address and postal code fields, the system maps the input address string to a geographic information feature code and matches it against address features in the postal code database. If the matching degree is lower than a preset threshold of 0.90, the system determines that the input item contains an implicit negative event, halts the automatic filling of that field in place, marks the field as pending confirmation, and requests user confirmation via a floating interactive window. In the event of manual intervention or secondary confirmation, the system employs a segmented validation compensation strategy when processing complex web forms containing numerous logical validation rules. For fill boxes with interdependent relationships in the form, the system parses the input data into a data stream with temporal correlation characteristics. When the system determines that the validity index of the input data is approaching a critical state, the system dynamically adjusts the correlation logic to reduce the computational load of the model processing and eliminate execution deviations caused by latency. The system records the anonymized interaction log of the entire filling process, stores only the input and output status fingerprint information within the current session period, and ensures that all cached data related to the current user information is automatically cleared after the filling process ends.

[0041] Example 5: Before actual deployment, the current web form intelligent filling system executes an offline parameter calibration procedure to establish an input benchmark reference table that meets business compliance requirements. In order to construct an initial benchmark with a high generalization boundary without interfering with the real-time operation of the front-end form filling business, the system uses an offline collection module deployed on the back-end management server to acquire all historical samples. The specific functions and roles of the offline collection module are as follows: as the core component of distributed historical message log filtering and formatting cleaning, it is responsible for periodically polling or asynchronously capturing all compliant filling messages generated by historical transactions through a dedicated database connection API, filtering out dirty data in the messages, and structuring and aligning the multimodal raw input stream according to timestamps, thereby providing a statistical sample source for system calibration. Through the offline acquisition module, the system can obtain historical valid data samples corresponding to the target web page form, select 500 complete sample records that have passed the backend verification, extract the input frequency distribution of 10 basic fields such as name, address, and date, and the time correlation features between each field. The system calculates the average input time and standard deviation of each field, and solidifies the obtained statistical data into the input rate model of each field, which serves as a quantitative benchmark for judging whether the input content conforms to the normal operation mode during subsequent real-time filling. During the automated filling of web page form data, the system establishes a real-time monitoring link for the interaction process of each filling box. The system traverses the document object model structure of the target form every 20ms to detect whether the attributes of the filling boxes have changed, and counts the switching frequency of users between each filling box. When the system detects that the input data features deviate from the preset feature distribution in the benchmark reference table by more than 30%, the system triggers the adaptive parameter correction logic. The system calculates and updates the feature decoding weight parameters of the multimodal large language model in real time based on the field density of the current page form and the contextual correlation of the filling boxes.

[0042] The system is configured with cross-validation rules based on the completeness and logical relevance of field input. By comparing input information from related fields on the same page, the system verifies for logical inconsistencies between input items. In scenarios involving ID document type and document number format verification, the system matches address information with geographic information feature codes. If the matching degree between the input address feature code and the document's location feature code is lower than a preset threshold of 0.90, the system determines that the data item has a logical consistency conflict. At this point, the system triggers a blocking process, marking the field status as pending correction and sending a control command containing the field identifier and a description of the conflict to the interaction entry point. This drives the user interface to guide the user to correct the error through highlighting and explanation of the anomaly. The system then completes the validity verification process. Previously, the system prohibited submitting the content of this field to the business server. The web form processing logic has self-correcting capabilities. When the system detects more than three consecutive data verification failures caused by network transient fluctuations, the system automatically increases the feature dimension of the input information entity feature vector from the basic 128-dimensional semantic space to 256 dimensions. By increasing the spatial information redundancy, the system improves the alignment accuracy of the model with the web front-end verification rules under complex working conditions. The system stores the key parameters in the above processing process, including the input sampling frequency, similarity judgment threshold, and logical consistency detection window, in the local cache to ensure that all sensitive processing links are executed in a closed loop within the terminal device. It is strictly prohibited to transmit the original data stream containing user identity information to the outside.

[0043] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0044] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for intelligent form filling on web pages based on a multimodal large language model, characterized in that, Includes the following steps: Step S1: Inject a floating interactive assistant into the target webpage through a browser plugin to serve as a floating interactive entry point. Then, parse the document object model structure data of the target webpage, establish a mapping relationship between form fill boxes and field attributes, and capture node changes in the document object model structure data of the target webpage through the document object model change listening interface to update the mapping relationship. Step S2: Receive multimodal input signals, call the fusion multimodal interaction module to perform cross-modal feature normalization and temporal alignment on the multimodal input signals, and call the multimodal large language model to perform joint visual and semantic parsing on the multimodal input signals to extract structured information entity sets; Step S3: Extract the field data types, length limits, required attributes, regular expression rules, enumeration value sets, numerical ranges, field dependencies, and cross-field logic rules of the form input boxes, and perform hierarchical mapping to classify the field data types, length limits, and required attributes into the first-level basic level, the regular expression rules, enumeration value sets, and numerical ranges into the second-level complex level, and the field dependencies and cross-field logic rules into the third-level field level. This process constructs multi-level deterministic feature boundary vectors and transforms the structured information entity set into a high-dimensional semantic feature vector. Step S4: Obtain the semantic vector of the form fill box. The semantic vector is obtained by extracting and fusing the semantic connotation, structural attributes, rule constraints, and topological context information of the form fill box, and then mapping it to a unified high-dimensional numerical space using an embedding encoding technique that is homologous to the multimodal large language model. Calculate the dot product of the high-dimensional semantic feature vector and the form fill box semantic vector to generate an intermediate semantic association matrix. Apply a conditional masking operator to the intermediate semantic association matrix based on the multi-level deterministic feature boundary vectors to perform masking filtering and generate an alignment probability matrix. First, select the feature channel with the highest alignment probability for single mapping binding, thereby formatting and generating the pre-filled result of the entire form in memory. The pre-filled result is used as an intermediate review object and input into the subsequent review process. After the form fill box is pre-filled, a global review process is triggered in the sandbox created by the browser plugin. The corresponding control flow branch is determined through dual-path validation logic. The final deterministic data corrected or confirmed by the dual-path validation logic is output as the final fill result and formally written and input into the corresponding form fill box through the browser plugin's built-in automation interface.

2. The intelligent form filling method for web pages based on a multimodal large language model according to claim 1, characterized in that, The process of determining the final filling result based on the alignment probability matrix in step S4 also includes a feedback convergence stage: Step S41, when the alignment probability corresponding to the alignment probability matrix is ​​lower than the preset alignment probability threshold, and the attribute of the corresponding structured information entity satisfies the required attribute, the local topological feature components of the hierarchical belonging node and the context adjacent node of the form filling box in the current document object model tree structure are retrieved, and the local topological feature components are directly superimposed on the high-dimensional semantic feature vector of the structured information entity.

3. The intelligent form filling method for web pages based on a multimodal large language model according to claim 1, characterized in that, The process of parsing the document object model structure data and updating the mapping relationship in step S1 specifically includes: parsing the document object model structure data of the target webpage, capturing the addition, deletion and attribute modification status of each node in the document object model structure data of the target webpage through the document object model change monitoring interface, and triggering the re-parsing process of the document object model structure data to update the mapping relationship.

4. The intelligent form filling method for web pages based on a multimodal large language model according to claim 1, characterized in that, Step S2 includes the following sub-steps: Step S21, receiving a multimodal input signal composed of text, speech, image, and form screenshot; Step S22, converting the speech in the multimodal input signal into natural language text, and calling a multimodal large language model to perform joint visual and semantic analysis on the image and form screenshot in the multimodal input signal, extracting the text fields and handwritten content, resolving the omissions and ambiguous expressions in the natural language text, and outputting a structured information entity set.

5. The intelligent form filling method for web pages based on a multimodal large language model according to claim 1, characterized in that, Step S3 includes the following sub-steps: Step S31, discretize the document object model structure data and extract the field data types, length limits, required attributes, regular expression rules, enumeration value sets, numerical ranges, field dependencies, and cross-field logic rules corresponding to the form fill boxes; Step S32, perform hierarchical mapping processing on the extraction results of Step S31, so that the field data types, length limits, and required attributes are classified into the first level of the basic level, the regular expression rules, enumeration value sets, and numerical ranges are classified into the second level of the complex level, and the field dependencies and cross-field logic rules are classified into the third level of the field level; Step S33, continue to perform vectorization construction processing on the hierarchical mapping results of Step S32 to obtain multi-level deterministic feature boundary vectors, and at the same time transform the structured information entity set into a high-dimensional semantic feature vector.

6. The intelligent form filling method for web pages based on a multimodal large language model according to claim 2, characterized in that, Step S4 also includes the following sub-steps: Step S42, if the text length of the corresponding structured information entity exceeds the corresponding length limit and the data format of the structured information entity does not meet the corresponding field data type, the weight value of the corresponding matching channel in the alignment probability matrix is ​​set to 0.

7. The intelligent form filling method for web pages based on a multimodal large language model according to claim 1, characterized in that, It also includes the following steps: Step S5: For form fill boxes with alignment probability lower than the preset alignment probability threshold, output a secondary interactive prompt, and control the form page of the target webpage to complete the data submission based on the received confirmation feedback signal.

8. The intelligent form filling method for web pages based on a multimodal large language model according to claim 2, characterized in that, Step S41 includes the following sub-steps: Step S411, extract the parent node features, child node features, and semantic attributes of the form fill box in the document object model tree, as the local topological feature components of the hierarchical belonging node and the contextual adjacent nodes.

9. The intelligent form filling method for web pages based on a multimodal large language model according to claim 1, characterized in that, After the interactive assistant is deployed in step S1, the following steps are continuously executed throughout the entire runtime of the target webpage: Step S6, during the runtime of the target webpage, the interactive assistant is kept in a floating interactive state and keeps in real-time linkage with the form page. When the target webpage is redirected or asynchronously refreshed, the parsing process of the new form page is triggered and the mapping relationship is updated.

10. A web page form intelligent filling system based on a multimodal large language model, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor calls and executes the computer program in the memory to implement the intelligent form filling method for web pages based on a multimodal large language model as described in any one of claims 1-9.

Images