An intelligent form filling method and system based on multi-modal input and dynamic configuration

Abstract

The application relates to an intelligent form filling method and system based on multi-modal input and dynamic configuration, and belongs to the field of digital processing.The method comprises the following steps: constructing a dynamic business model; receiving input data of a user to perform multi-modal intent recognition, weighting and fusing text intent and attachment features, obtaining a target business type, and loading a corresponding dynamic business model; constructing a multi-modal enhanced context; extracting structured data from user text and attachment structured information and performing verification; when it is detected that a field or an attachment is missing, external data sources are actively pulled to obtain the missing content for supplementation; answering information and a form are dynamically generated according to the data structure of the business model, and the passed data is automatically filled into the corresponding fields of the form after extraction and verification; a submission operation is performed according to the interface configuration in the business model, and after the submission is completed, a conversation flow is transferred, and a new form filling scene is selected or the conversation is ended. The application realizes intelligent form filling with high universality, high precision and easy integration.

Description

Technical Field

[0001] This invention belongs to the field of digital processing, and in particular relates to an intelligent form filling method and system based on multimodal input and dynamic configuration. Background Technology

[0002] In enterprise digital office scenarios, form filling (such as travel expense reimbursements, purchase requests, and work order entry) is a frequent and tedious operation. Existing automated form filling technologies have mainly evolved into the following three categories: 1. OCR / image recognition-based form filling technology Key information (such as amount, date, merchant) is extracted from invoice and document images using optical character recognition (OCR) technology, and then filled into predefined fields using template matching or simple natural language processing (NLP) technology.

[0003] The invention patent with application publication number CN117807967A discloses a financial reimbursement method based on OCR intelligent form filling, and the invention patent with application publication number CN119830874A discloses an intelligent form filling method and device based on expense reimbursement business. The technical solutions of the above two invention patents both adopt form filling technology based on OCR / image recognition.

[0004] OCR / image recognition-based form filling technology has the following drawbacks: it can only handle visual-to-text conversion and cannot understand users' natural language instructions; it cannot handle complex business scenarios that require logical judgment, association with external knowledge bases, or implicit information; it has a low recognition rate for non-standard invoices or blurry images and lacks integration with the dialogue context.

[0005] 2. Form filling technology based on rule engine / QA guidance By using predefined decision trees, knowledge graphs, or rule engines, users are guided step-by-step to answer specific questions in a question-and-answer format, thereby filling in form fields.

[0006] The invention patent with application publication number CN120181054A discloses a dynamic form generation method based on multi-turn dialogue, and the invention patent with application publication number CN116776840A discloses an intelligent form filling method based on text generation. The technical solutions of the above two invention patents both adopt form filling technology based on rule engine / QA guidance.

[0007] The form-filling technology based on rule engines / QA guidance has the following drawbacks: the interactive experience is rigid, and users must follow the system's preset path; the rule maintenance cost is extremely high, and a large number of rules need to be reconfigured for each new business type; it lacks the ability to deeply understand semantics and cannot handle the complex intentions freely expressed by users and multimodal inputs (such as directly uploading images).

[0008] 3. Form filling technology for fine-tuning large models (large models) based on specific scenarios: For specific business scenarios (such as business travel and customer service work orders), a large amount of historical data is collected to perform full or incremental fine-tuning on the base model, enabling it to have the ability to extract data in that scenario.

[0009] The invention patent with application publication number CN121660630A discloses an intelligent form filling method and system based on a large model for travel expense reimbursement forms, and the invention patent with application publication number CN120632106A discloses an intelligent form filling fine-tuning system and method based on a large model. The technical solutions of the above two invention patents both adopt the form filling technology of fine-tuning the large model (large model) based on specific scenarios.

[0010] The form-filling technique of fine-tuning a large model (large model) based on a specific scenario has the following drawbacks: Poor versatility: Each new service requires retraining or fine-tuning, resulting in high computing costs and long cycles.

[0011] Knowledge lag and compliance risks: Fine-tuning models may struggle to obtain the latest corporate policies and regulations in real time, potentially leading to data extraction violations.

[0012] Missing context: User input is often processed in isolation without fully integrating user profiles (job title, permissions), spatiotemporal information (time, location), and unstructured attachments, leading to decreased accuracy under complex business logic.

[0013] In summary, the existing technology has the following problems: (1) Poor business adaptability (high coupling): Existing solutions often hardcode the form structure and validation logic in the code or model, lacking a general abstraction layer, resulting in extremely high development and maintenance costs for new business.

[0014] (2) Fragmentation of multimodal understanding: Existing systems struggle to semantically integrate users' natural language commands with uploaded attachments or files obtained from external systems. For example, when a user uploads an invoice image, the system often needs to trigger OCR separately, rather than understanding it naturally within the conversation.

[0015] (3) Data silos and passive processing: The existing form filling system cannot actively call up historical files in the enterprise's internal image system (EDMS) and ERP system. Users must manually download, upload, and re-identify, which is a cumbersome process, and the system lacks the ability to proactively detect missing vouchers and automatically complete them.

[0016] (4) Insufficient flexibility in backend integration: Existing systems typically only support a single backend interaction mode (such as only supporting REST API), making it difficult to simultaneously support modern front-end separation architecture and traditional monolithic architecture (such as needing to redirect to old system pages), thus limiting the scope of system promotion. Summary of the Invention

[0017] In view of the shortcomings of the prior art, the purpose of the invention is to provide an intelligent form filling method and system based on multimodal input and dynamic configuration. By decoupling business logic and code, enhancing multimodal context awareness, actively pulling external data and adapting to heterogeneous backends, it achieves highly versatile, highly accurate and easily integrated intelligent form filling.

[0018] In a first aspect, the present invention proposes an intelligent form-filling method based on multimodal input and dynamic configuration, comprising: S1, define a standardized business model and its key attributes and attribute descriptions, and build a dynamic business model; S2, receive user input data, perform multimodal intent recognition on the input data, obtain text intent and attachment features, perform weighted fusion on the text intent and attachment features, obtain the final target business type and load the corresponding dynamic business model; S3 constructs a multimodal enhanced context, including: user profile layer, spatiotemporal environment layer, knowledge enhancement layer, attached structured information and structure guidance layer; S4. Based on the enhanced context, the large language model is invoked to extract structured data from the user's text and attachment structured information, establish the association between fields and attachments, verify the compliance and completeness of fields and attachments, and when a field or attachment is detected to be missing, the missing content is actively retrieved from an external data source to supplement it until the field and attachment are complete. S5. Dynamically generate answer information and forms based on the data structure of the business model, automatically fill the corresponding fields of the form with the data extracted and verified in step S4, and display the form in a visual UI interface. S6, after receiving the user's confirmation submission instruction, executes the submission operation according to the interface configuration in the business model. After the submission is completed, the session flow is performed, allowing users to choose to enter a new form scenario or end the session.

[0019] Furthermore, in S1, the business model includes: business system, business metadata, data model, credential and attachment mapping rules, external data source configuration, interface configuration, permission model, and post-processing actions.

[0020] Further, in S2, multimodal intent recognition is performed on the input data, including: Perform semantic parsing on the input text and speech; The uploaded attachment text content is parsed, attachment entity features are extracted, and the attachment entity features are matched with the voucher and attachment mapping rules in the business model to help determine the user's intent.

[0021] Furthermore, in S3, the user profile layer includes: the current user's organizational information, position, responsibilities, work location, and reporting and mentoring relationships; The spatiotemporal environment layer includes: the current system time and the user's current location information; and the current time based on the scene's system time. The knowledge enhancement layer includes: retrieving relevant constraint rules, filling examples, or policy fragments from the enterprise knowledge base based on the target business type, user input, and attachment keywords; The structured information of the attachments includes: converting the parsed key information of the attachments into structured text blocks and embedding them in the Prompt; The structure guidance layer includes: the data structure definition and verification rules of the target business model.

[0022] Furthermore, in S4, when a detection field or attachment is missing, and an external data source is configured in the business model, query parameters are automatically generated based on the current user ID, time range, and business type; the external data source is actively called via API to retrieve and download relevant files; the downloaded files are parsed again to obtain the missing content and supplement it into the context.

[0023] Furthermore, in S4, if no external data source is configured or the retrieval fails, a natural language prompt is generated to encourage the user to manually upload the missing content.

[0024] Furthermore, in S4, the triggering conditions for actively pulling external data sources include the following two types: Configure a scheduled synchronization mechanism to automatically fetch specified external data sources and push them to the forms to be filled at regular intervals; or fetch the full external data source when user authorization is detected.

[0025] Furthermore, in S5, the form supports displaying previews of associated attachments next to the corresponding fields, and allows users to replace or remove the attachment previews.

[0026] Furthermore, in S5, the following interactive functional interfaces are dynamically generated based on the data structure of the business model: intelligent navigation, intelligent Q&A, and intelligent data query. Intelligent navigation: Matches the business function menus or page routes of the business model. When a match is successful, it directly jumps to the corresponding function page, displays relevant data dashboards, or provides quick operation access. Intelligent Q&A: Match business model information, search corporate documents or policies, and extract or answer questions based on user profiles and search results; Intelligent Question Data: Matches business model information with external data sources and business data to perform semantic understanding of question data and output reports.

[0027] Furthermore, in S6, the step of performing the submission operation according to the interface configuration in the business model includes: For a front-end and back-end separated model, construct an HTTP request, call the back-end API, and simultaneously upload the attachment stream or pass the attachment reference ID; For front-end and back-end not separated, a hidden form can be constructed or a browser redirect can be performed directly to redirect the user to the traditional system on the other end.

[0028] A second aspect of the present invention proposes an intelligent form-filling system based on multimodal input and dynamic configuration, comprising: The business model configuration center is used to define standardized business models and their key attributes and attribute descriptions, and to build dynamic business models. A multimodal intent recognition engine is used to perform multimodal intent recognition on user input data, obtain text intent and attachment features, perform weighted fusion of the text intent and attachment features to obtain the final target business type and load the corresponding dynamic business model; The Enhanced Context module is used to construct a multimodal enhanced context, including: a user profile layer, a spatiotemporal environment layer, a knowledge enhancement layer, attached structured information, and a structured guidance layer; The large model orchestration service module is used to extract structured data from user text and attachment structured information based on the enhanced context, call the large language model, establish the association between fields and attachments, verify the compliance and completeness of fields and attachments, and when a field or attachment is detected to be missing, the external data adapter actively pulls external data sources to obtain the missing content to supplement it until the field and attachment are complete. The session state manager is used to dynamically generate answer information and forms based on the data structure of the business model, automatically fill the corresponding fields of the form with the extracted and validated data, and display the form in a visual UI interface; it also maintains the form filling status, filled data, attachment list and context history of the current session. The backend adaptation gateway is used to execute the submission operation according to the interface configuration in the business model after receiving the user's confirmation submission instruction. After the submission is completed, the session flow is performed, allowing users to choose to enter a new form scenario or end the session.

[0029] Furthermore, the multimodal intent recognition engine integrates keyword retrieval, vector retrieval, a small model classifier, and a multimodal attachment parser.

[0030] The beneficial effects of this invention are as follows: The method and system described in this invention are intelligent form filling methods and systems based on dynamic business models and multimodal context enhancement. By decoupling business logic from code, enhancing multimodal context awareness, actively pulling external data, and adapting to heterogeneous backends, it achieves highly versatile, highly accurate, and easily integrated intelligent form filling. This invention is based on the same dynamic business model architecture and also supports an intelligent navigation mode, which uses semantic understanding to match the user's input business intent and automatically redirects to the corresponding function page or data dashboard, serving as a supplementary interactive method for intelligent form filling.

[0031] 1. Employing multimodal fusion intent recognition significantly improves intent recognition accuracy. This invention employs multimodal fusion intent recognition, combining text semantics with attachment entity features, to achieve intent-assisted determination based on the mapping of attachment content entity features with business model credentials, significantly improving the accuracy of intent recognition.

[0032] In the intent recognition stage, the multimodal complementarity of this invention not only analyzes the text but also parses the content of user-uploaded attachments in real time. It uses entity features in the attachments (such as invoice category and contract amount) to assist in judging business intent and achieves accurate routing in scenarios where only images / files are sent without text. For example, the text may be ambiguous (e.g., "reimburse expenses"), but the attachment is clear (e.g., "restaurant invoice"). Combining the two eliminates ambiguity and improves the robustness of recognition under noisy data or ambiguous expressions.

[0033] 2. Employing an external data source proactive retrieval strategy, the user operation is extremely simple, achieving seamless form filling. This invention adopts an external data source proactive retrieval strategy based on a business model to automatically obtain missing invoices from the imaging system / ERP, and realizes automatic file acquisition and parsing based on missing voucher detection and external system configuration linkage.

[0034] This invention enables proactive human-computer interaction, shifting from traditional passive reception to proactive service. It leverages inter-system interconnectivity to fill gaps in user operations, significantly reducing the burden of manual file uploading and searching, and enhancing user experience. Compared to the traditional passive waiting for user uploads, this invention automatically triggers API calls to external systems such as imaging systems and ERP systems when missing data is detected, based on the required attachments and external data source configurations defined in the business model. This retrieves, parses, and populates historical files, achieving seamless form filling.

[0035] 3. Employ a field-attachment structured association mapping to achieve strong consistency between data and vouchers. This invention employs a field-attachment structured association mapping technology, which simultaneously returns data values ​​and corresponding attachment IDs when outputting a large model, thereby achieving automatic association mapping between form fields and unstructured attachments during the large model extraction process.

[0036] This invention embeds field-attachment mapping rules into the Prompt, forming a structured association mapping between fields and attachments. This allows the large model to not only extract data but also automatically establish logical bindings between data values ​​and supporting documents, ensuring that each key data field has a corresponding chain of evidence. This invention also generates structured constraints on the large model output through schema constraints, ensuring that each key data field has a corresponding chain of evidence (attachments), meeting financial / audit compliance requirements and reducing compliance risks.

[0037] 4. Employ a dynamic business model to achieve high business versatility and agile configuration. This invention employs a dynamic business model, decoupling business logic from the code. Configuration-driven development shortens the new business launch cycle from weeks to hours, reduces code coupling, and adapts to the rapid iteration needs of enterprise businesses.

[0038] This invention proposes a decoupled architecture driven by a dynamic business model, which is a multi-dimensional dynamic business model that includes business metadata, data model, voucher mapping, external data source, and interface configuration. This allows new business to be launched without modifying the code or retraining the model; simply updating the model configuration is sufficient. It supports the dynamic business model structure of voucher mapping and external data source configuration and its application in intelligent form filling.

[0039] This invention proposes a decoupled architecture for interaction methods and intent navigation. It is a multi-dimensional dynamic business model that includes business metadata, data models, credential mapping, external data sources, and interface configurations. This model enables the integration or fusion of multiple intelligent agent functions without requiring code modification or model retraining when adding new services. This invention realizes the application of digital humans based on dynamic business models and multi-agent fusion in information systems.

[0040] 5. Seamless heterogeneous system compatibility This invention employs a backend adaptation gateway to distinguish between API calls and redirects. The adapter pattern of this invention enables a single frontend intelligent form-filling system to be compatible with both modern microservice architectures and traditional monolithic architecture backends, thus resolving system integration pain points. Attached Figure Description

[0041] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. It is obvious that the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings.

[0042] Figure 1 This is a flowchart of an intelligent form filling method based on multimodal input and dynamic configuration according to an embodiment of the present invention; Figure 2This is a UI diagram of the form in an embodiment of the present invention; Figure 3 This is an architecture diagram of an intelligent form-filling system based on multimodal input and dynamic configuration, according to an embodiment of the present invention. Detailed Implementation

[0043] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0044] Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts disclosed in this invention.

[0045] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The terms "installed," "connected," and "linked" should be interpreted broadly; for example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0046] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of methods and systems consistent with some aspects of the invention as detailed in the appended claims.

[0047] like Figure 1 As shown, this invention proposes an intelligent form-filling method based on multimodal input and dynamic configuration, including: S1 defines a standardized business model and its key attributes and attribute descriptions, and constructs a dynamic business model.

[0048] In this invention, the business model includes: business system, business metadata, data model, credential and attachment mapping rules, external data source configuration, interface configuration, permission model, and post-processing actions.

[0049] The following sections will explain each item in the business model.

[0050] Business System: Business system name, single sign-on configuration, business description (for intent recognition, specifically including the functional positioning, scope of application, core business process overview and relationship with other systems of the business system), and API base URL.

[0051] Business metadata: business system to which it belongs, business name, alias set (for intent matching), business description text, permission code (for access control); relationships between businesses (association constraints, etc., used to control the completeness of input, such as expense reports must have at least one line of expense information, and junior mentors can guide a maximum of one new employee).

[0052] Data Model: A collection of field definitions, including business scope, field identifier, label, data type, constraint rules (required / optional, format regular expression, enumeration value, logical dependencies between fields), aliases (for intent understanding), attribute descriptions (for intent understanding), default values, and data sources (including user attributes, user sessions, attachments, etc.; non-user session fields cannot be extracted from the session).

[0053] Voucher / Attachment Mapping Rules: Define the association between business fields and attachment types (e.g., the transportation expense field should be associated with an invoice image attachment).

[0054] External data source configuration: Define the external system interfaces required for this business (such as: imaging system API, ERP voucher interface), including interface address, authentication method, and query parameter template.

[0055] Interface configuration: API definition (URL, Method, Headers, Payload structure) or redirection strategy.

[0056] Permission model: role mapping table and access control list.

[0057] Post-processing actions: Define the behavior after successful submission.

[0058] The invention proposes a multi-dimensional dynamic business model that includes business metadata, data model, credential mapping, external data source, and interface configuration. This model enables the integration or fusion of various intelligent agent functions without requiring code modification or model retraining for new business applications. The system can be deployed simply by updating the model configuration.

[0059] S2, Multimodal Intent Recognition and Form Routing: Receives user input data, performs multimodal intent recognition on the input data, obtains text intent and attachment features, performs weighted fusion of text intent and attachment features to obtain the final target business type and loads the corresponding dynamic business model.

[0060] S21, perform semantic parsing on the input text and speech.

[0061] Specifically, it receives user input data, including text, voice, and uploaded attachments (text / images / PDFs / Office documents, etc.).

[0062] S22, perform multimodal fusion intent recognition, determine the target business type and load the corresponding dynamic business model.

[0063] First, text analysis is performed: semantic analysis is conducted on the text / speech to extract keywords, etc. Then, the attached content is parsed and intent is assisted.

[0064] If the user uploads an attachment, the text content of the uploaded attachment is parsed, the entity features of the attachment are extracted, and the entity features of the attachment are matched with the voucher and attachment mapping rules in the business model to help determine the user's intent.

[0065] Specifically, the file content is parsed according to the format of the attachment, and then the key entities in the attachment (such as: invoice code, amount, invoice date, product name, contracting party) are extracted.

[0066] In this invention, parsing the file content according to the format of the appendix includes: If it's a text file, the content is read directly. If it's a PDF file, the text is extracted. If a page with a sufficient length of text (e.g., less than 10 characters) cannot be extracted, the corresponding page is converted into an image and OCR is performed. If it's audio or video, multimodal large model conversion is used.

[0067] The above-mentioned attachment parsing method can also adopt a multimodal understanding model, namely: a unified visual language model, or a combined architecture (OCR engine + text model + rule engine).

[0068] After extraction, the extracted attachment entity features are matched with the voucher fields in the business model to help determine the user's intent. For example, if a Didi trip receipt is uploaded, the system can infer that the intent may be transportation reimbursement.

[0069] Therefore, in the intent recognition stage, this invention not only analyzes the text, but also further parses the content of the attachments uploaded by the user in real time, and uses the entity features in the attachments (such as invoice category and contract amount) to reversely assist in judging the business intent, and achieves accurate routing in the scenario of "only sending pictures / files without speaking".

[0070] This invention employs a multimodal complementary approach of text and attachments. In cases where the text is ambiguous, the attachments provide clear and supplementary explanations. The combination of the two eliminates ambiguity and improves the robustness of recognition under noisy data or ambiguous expressions.

[0071] S23, Comprehensive Decision: Weighted fusion of text intent weight and attachment feature weight to determine the final target business type.

[0072] Specifically, the text intent weight and attachment feature weight are calculated based on the default data source configured in the dynamic business model in step S1 (the default data source weight is 5, and the non-default data source weight is 1 lower). This is used to handle the priority of the same attributes appearing in the direct input text or uploaded attachment extraction, thereby improving the accuracy of business type judgment.

[0073] For example, if the input text requests a 3-day leave and the attachment requests a 2-day leave, and the business model is configured with a higher weight for the input text than for the attachment, then the 3-day leave will be extracted; otherwise, the 2-day leave will be extracted.

[0074] S3 constructs a multimodal enhanced context, including: a user profile layer, a spatiotemporal environment layer, a knowledge enhancement layer, attached structured information, and a structure guidance layer.

[0075] Specifically, before data extraction, an enhanced context is dynamically constructed, including the following dimensions: The user profile layer includes: automatically acquiring and injecting the current user's organizational information, job position, responsibilities, work location, reporting and mentoring relationships, etc.

[0076] The spatiotemporal environment layer includes: automatically injecting the current system time (accurate to the minute) and the user's current location information; and the current time based on the scene's system time (including the lunar calendar, Buddhist calendar, etc.).

[0077] The Knowledge Enhancement Layer (RAG) includes: retrieving relevant constraint rules, filling examples, or policy fragments from the enterprise knowledge base (systems, manuals, guidelines) based on the target business type and user input / attachment keywords.

[0078] The structured information in the attachments includes: converting the key information of the attachments parsed in step S2 into structured text blocks and embedding them in the Prompt.

[0079] The structure guidance layer includes: the data structure definition (Schema) and validation rules for injecting the target business model.

[0080] It should be noted that the data structure definition and verification rules of the above target business model can be applied to subsequent compliance and integrity verification.

[0081] S4, based on enhanced context, calls the large language model to extract structured data from user text and attachment structured information, establishes the association between fields and attachments, verifies the compliance and completeness of fields and attachments, and when a field or attachment is detected to be missing, it actively pulls external data sources to obtain the missing content to supplement it until the field and attachment are complete.

[0082] S41, Data Extraction and Association By embedding field-attachment mapping rules in the Prompt, the large language model can not only extract data, but also automatically establish logical bindings between data values ​​and supporting evidence, ensuring that each key data field has a corresponding chain of evidence to support it.

[0083] This invention extracts structured data from user text and attachment structured information, establishes field-attachment relationships, and returns both the data value and the corresponding attachment ID when outputting the large model. For example, transportation fee: 600 yuan; associated attachment ID_001.

[0084] This invention constrains the output of a large model through schema constraints, ensuring that each key data field is supported by a corresponding chain of evidence (attachments), thereby meeting financial / audit compliance requirements and reducing compliance risks.

[0085] S42, Compliance and Integrity Verification Check for missing required fields and required attachments. For example, a contract is required for amounts greater than 500 yuan. If not, it indicates that key documentation is missing.

[0086] S43, Actively trigger external data acquisition If a key credential (required field or required attachment) is detected to be missing, and an external data source (such as an imaging system) is configured in the business model, query parameters will be automatically generated based on the current user ID, time range, and business type.

[0087] By actively calling external data sources (such as a specified imaging system / ERP system) via API, relevant historical invoices or documents are retrieved and downloaded; the downloaded files are then parsed to obtain missing content and supplement it into the context.

[0088] In this invention, external data sources can also be: bank transaction interfaces (automatically obtaining payment vouchers), airline / hotel system interfaces (automatically obtaining booking itineraries), business platforms (such as travel management systems), third-party tax platforms (automatically obtaining input invoices), and other information systems. The configuration of external data sources is based on user needs and will not be elaborated further here.

[0089] It should be noted that if no external data source is configured or the search fails, a natural language prompt will be generated to guide the user to manually upload the missing content.

[0090] S43, Iterative Interaction If any key credentials are still missing, a prompt will be generated to guide the user to supplement them. If the user continues to input or upload, steps S3 and S4 will be executed iteratively until all constraints are met.

[0091] In this invention, the triggering conditions for actively fetching external data sources, in addition to detecting missing fields or attachments, further include the following two: (1) Configure a timed synchronization mechanism to automatically retrieve specified external data sources and push them to the forms to be filled at regular intervals. For example, automatically retrieve yesterday's invoices and push them to the list of forms to be filled every morning.

[0092] (2) Once user authorization is detected, the external data source is fully retrieved.

[0093] By proactively fetching external data sources, the system no longer passively waits for users to upload data. Instead, based on the required attachments defined in the business model and the configuration of external data sources, it automatically triggers API calls to external systems such as imaging systems and ERP systems when missing data is detected, retrieves, parses, and populates historical files, thus achieving seamless form filling.

[0094] S5: Dynamically generate answer information and forms based on the data structure of the business model, automatically fill the corresponding fields of the form with the data extracted and verified in step S4, and display the form in a visual UI interface.

[0095] This step involves dynamic rendering and confirmation on the front end. Specifically, the front end dynamically generates the answer information and form UI based on the data structure of the business model, automatically fills the corresponding fields in the form with the data extracted and verified in step S4, and displays the data visually.

[0096] In this invention, the form supports displaying a preview of the associated attachment next to the corresponding field, and allows users to replace or remove the attachment preview.

[0097] like Figure 2 As shown, previews of associated attachments (such as invoice thumbnails) are displayed next to the corresponding fields, allowing users to replace or remove them. Figure 2 The currently selected preview thumbnail of Attachment 2.

[0098] It should be noted that, in addition to the automatic association method by the large model mentioned above, this invention also supports the following two association methods for forms and attachments: (1) Drag and drop interaction: The front end displays all parsed attachment cards. Users can manually drag and drop cards to the corresponding form fields. The system records the association relationship to realize the association.

[0099] (2) User input or selection: Users can directly input or select business information in the rendered form. For example, if the business metadata defines business trip types as foreign affairs and training, users can directly select the business trip type as training in the rendered form to achieve association.

[0100] In addition to using intelligent form filling as the core interaction method, this invention, based on the same dynamic business model architecture, further supports the following interaction entry points and navigation methods according to the data structure of the business model, dynamically generating corresponding interactive function interfaces: intelligent navigation mode, intelligent question-and-answer mode, and intelligent data query mode. Users describe their business intent by inputting natural language, such as "I want to claim travel expenses" or "Check contract progress," and the system calls a semantic understanding model to parse the user's intent.

[0101] Intelligent navigation: By semantic understanding, it matches the user's business intent with the business function menu or page route of the business model. When the match is successful, it automatically jumps to the corresponding function page, displays relevant data dashboards, or provides quick operation entry points, serving as a supplementary interaction method to intelligent form filling.

[0102] Intelligent Q&A: Match business model information, search for company documents or policies, and extract or answer questions based on user profiles and search results.

[0103] Intelligent Question Data: Matches business model information with external data sources and business data to perform semantic understanding of question data (data statistics, analysis) and report output.

[0104] S6, Adaptive Backend Submission and Session Flow: After receiving the user's confirmation submission instruction, the submission operation is executed according to the interface configuration in the business model. After the submission is completed, the session flows, allowing users to choose to enter a new form scenario or end the session.

[0105] Specifically, the commit operation is performed according to the interface configuration in the business model, including: (1) For the front-end and back-end separation mode (front-end and back-end separation adaptation): construct an HTTP request, call the back-end API, and upload the attachment stream or pass the attachment reference ID at the same time.

[0106] (2) For the non-separated front-end and back-end mode (non-front-end and back-end separation adaptation): construct a hidden form or directly execute browser redirection to jump the user to the traditional system on the other end.

[0107] like Figure 3 As shown, this invention proposes an intelligent form filling system based on multimodal input and dynamic configuration, including: a business model configuration center 100, a multimodal intent recognition engine 200, an enhanced context module 300, a large model orchestration service module 400, an external data adapter 500, a session state manager 600, and a backend adaptation gateway 700.

[0108] Specifically, the Business Model Configuration Center 100 is used to define standardized business models and their key attributes and attribute descriptions, build dynamic business models, and store and manage business models.

[0109] The multimodal intent recognition engine 200 is used to perform multimodal intent recognition on user input data, obtain text intent and attachment features, perform weighted fusion of text intent and attachment features to obtain the final target business type and load the corresponding dynamic business model.

[0110] In this invention, the multimodal intent recognition engine 200 integrates keyword retrieval, vector retrieval, small model classifier, and multimodal attachment parser.

[0111] Integrated Keyword Retrieval: Based on the set of names and aliases defined in the business model, keyword matching is performed on user input to quickly recall candidate business types.

[0112] Vector retrieval: Using a pre-trained text embedding model, user input is mapped to business descriptions in the business knowledge base as high-dimensional vectors, and cosine similarity is calculated for semantic-level matching.

[0113] Small model classifier: refers to a business classification model that has been lightly tuned and is designed for a specific business type of an enterprise.

[0114] Keyword retrieval, vector retrieval, small model classifier, and multimodal attachment parser can be used individually or in combination (e.g., keyword and vector retrieval for coarse ranking, and small model classifier for fine ranking) to improve the accuracy and recall of intent recognition.

[0115] The Enhanced Context Module 300 is used to construct a multimodal enhanced context, including: a user profile layer, a spatiotemporal environment layer, a knowledge enhancement layer (RAG), attached structured information, and a structured guidance layer.

[0116] Among them, the RAG knowledge base service connects with enterprise Wiki / documents, providing vector indexing and semantic retrieval, and can pre-generate knowledge indexes that match job positions based on business scale.

[0117] The large model orchestration service module 400 is responsible for prompt assembly, large model invocation, and result parsing. Specifically, based on the enhanced context, the large model orchestration service module 400 calls the large language model to extract structured data from user text and attachment structured information, establishes associations between fields and attachments, and verifies the compliance and completeness of fields and attachments. When a field or attachment is detected to be missing, the external data adapter 500 actively retrieves the missing content from external data sources to supplement it until the fields and attachments are complete. The external data adapter 500 is responsible for interfacing with external data sources such as imaging systems and ERP systems, and executing the active retrieval logic.

[0118] The Session State Manager 600 is used to dynamically generate answer information and forms based on the data structure of the business model. It automatically fills the corresponding fields of the form with the extracted and validated data and displays the form in a visual UI interface. It also maintains the form filling status, filled data, attachment list and context history of the current session.

[0119] The backend adapter gateway 700 is used to execute the submission operation according to the interface configuration in the business model after receiving the user's confirmation submission instruction. After the submission is completed, the session flow is performed, allowing users to choose to enter the new form filling scenario or end the session.

[0120] In this invention, the backend adapter gateway encapsulates the calling logic for RESTful APIs, Web services, or HTTP redirects.

[0121] The intelligent form filling method and system based on multimodal input and dynamic configuration of the present invention will be described below with reference to specific embodiments.

[0122] Example 1: Travel expense reimbursement (including automatic retrieval of historical invoices) Scenario description: Employee Li Si says in the conversation, "Could you please report my business trip to Hangzhou last month for the project?" Furthermore, Li Si did not upload any attachments.

[0123] Implementation process: 1. Intent Recognition: Text analysis yields the words "report" and "travel", identifying the intent as travel reimbursement, and then loading the travel reimbursement model.

[0124] 2. Context Construction and Missing Detection: The large model was extracted last month; the location is Hangzhou. According to the requirements of the travel expense reimbursement model, transportation and accommodation expenses are mandatory and must be accompanied by invoice attachments. The current status is determined to be: missing required attachments.

[0125] 3. Actively pull external data: The system checked the model configuration and found the following configuration: source_system:"CORP_IMAGE_SYSTEM".

[0126] System-constructed query: User:LiSi, Date:LastMonth, Type:Invoice.

[0127] By calling the image system API, two invoices scanned and uploaded by Li Si last month were retrieved: Inv_01 (Hangzhou bus, 50 yuan) and Inv_02 (Hangzhou hotel, 1200 yuan, 3 nights' stay, from June 2nd to 5th).

[0128] The system automatically performs OCR parsing on Inv_01 and Inv_02 to extract the amount, date, and number of nights of stay.

[0129] 4. Data fusion and verification: Contextual information: Li Si is the manager of the XX department, holding a management level 4 position, and works in Shanghai (user profile). He is on a business trip outside the province, and the invoice indicates that the trip took place in June (time information).

[0130] RAG Knowledge Base search results for financial regulations and travel regulations: Search for the reimbursement system for business trips outside the province to Hangzhou for four management positions in June.

[0131] Assuming the standard for business trips outside the province for positions at level four and above is 300 yuan per night, and the peak season standard for Hangzhou is June each year, which is the peak tourist season, the accommodation fee for department managers is increased by 200 yuan.

[0132] By merging context, RAG search results, and parsed information, the associated transportation cost was 50 yuan (Inv_01), and the accommodation cost was 1200 yuan (Inv_02). The accommodation standard was verified, and the large model determined that Li Si met the Hangzhou peak season accommodation standards and did not exceed financial management requirements.

[0133] 5. Front-end rendering and confirmation: The front-end generates a form, automatically fills in the amount, and displays thumbnails of Inv_01 and Inv_02 next to the fields. Prompt: "Two invoices have been automatically found. Are there any others?" 6. Submit: The user confirms "No more, submit". The system constructs a payload containing field data and a list of attachment IDs, and calls the backend API to complete the submission.

[0134] Example 2: User directly uploads attachments (no text). Scenario Description: The user directly uploaded an image of a "Didi Electronic Trip Receipt" (denoted as Attachment A) and a... Image of a "General Value Added Tax Invoice" (referred to as Attachment B).

[0135] Implementation process: 1. Intent Recognition (Attachment-Driven): Multimodal model image analysis: Attachment A identifies "Didi Chuxing" and "Amount 88 yuan"; Attachment A also identifies "Catering Service", "Amount 200 yuan", travel time, and the time when the business expense (catering) occurred.

[0136] Intent inference: Given the presence of transportation and dining characteristics, the case is identified as a reimbursement of daily expenses.

[0137] 2. Data extraction and association: The large model outputs transportation costs of 88 yuan (Appendix A) and catering costs of 200 yuan (Appendix B).

[0138] 3. Interactive completion: When a missing reason is detected, the system replies: "Transportation fee of 88 yuan and catering fee of 200 yuan have been identified. What is the reason?" 4. Completion: The user replies "customer hospitality", the system completes the reason, renders the form, and the user submits.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the protection scope of the present invention.

Claims

1. A smart form-filling method based on multimodal input and dynamic configuration, characterized in that, include: S1, define a standardized business model and its key attributes and attribute descriptions, and build a dynamic business model; S2, receive user input data, perform multimodal intent recognition on the input data, obtain text intent and attachment features, perform weighted fusion on the text intent and attachment features, obtain the final target business type and load the corresponding dynamic business model; S3 constructs a multimodal enhanced context, including: user profile layer, spatiotemporal environment layer, knowledge enhancement layer, attached structured information and structure guidance layer; S4. Based on the enhanced context, the large language model is invoked to extract structured data from the user's text and attachment structured information, establish the association between fields and attachments, verify the compliance and completeness of fields and attachments, and when a field or attachment is detected to be missing, the missing content is actively retrieved from an external data source to supplement it until the field and attachment are complete. S5. Dynamically generate answer information and forms based on the data structure of the business model, automatically fill the corresponding fields of the form with the data extracted and verified in step S4, and display the form in a visual UI interface. S6, after receiving the user's confirmation submission instruction, executes the submission operation according to the interface configuration in the business model. After the submission is completed, the session flow is performed, allowing users to choose to enter a new form filling scenario or end the session.

2. The intelligent form-filling method based on multimodal input and dynamic configuration according to claim 1, characterized in that, In S1, the business model includes: business system, business metadata, data model, credential and attachment mapping rules, external data source configuration, interface configuration, permission model, and post-processing actions.

3. The intelligent form filling method based on multimodal input and dynamic configuration according to claim 1, characterized in that, In S2, multimodal intent recognition is performed on the input data, including: Perform semantic parsing on the input text and speech; The uploaded attachment text content is parsed, attachment entity features are extracted, and the attachment entity features are matched with the voucher and attachment mapping rules in the business model to help determine the user's intent.

4. The intelligent form-filling method based on multimodal input and dynamic configuration according to claim 1, characterized in that, In S3, the user profile layer includes: the current user's organizational information, position, responsibilities, work location, and reporting and mentoring relationships; The spatiotemporal environment layer includes: the current system time and the user's current location information; and the current time based on the scene's system time. The knowledge enhancement layer includes: retrieving relevant constraint rules, filling examples, or policy fragments from the enterprise knowledge base based on the target business type, user input, and attachment keywords; The structured information of the attachments includes: converting the parsed key information of the attachments into structured text blocks and embedding them in the Prompt; The structure guidance layer includes: the data structure definition and verification rules of the target business model.

5. The intelligent form-filling method based on multimodal input and dynamic configuration according to claim 1, characterized in that, In S4, when a detection field or attachment is missing, and an external data source is configured in the business model, query parameters are automatically generated based on the current user ID, time range, and business type. The system actively calls external data sources via API to retrieve and download relevant files; the downloaded files are then parsed to obtain missing content and add it to the context.

6. The intelligent form-filling method based on multimodal input and dynamic configuration according to claim 1, characterized in that, In S4, the triggering conditions for actively pulling external data sources further include the following two: Configure a scheduled synchronization mechanism to automatically fetch specified external data sources and push them to the forms to be filled at regular intervals; or fetch the full external data source when user authorization is detected.

7. The intelligent form-filling method based on multimodal input and dynamic configuration according to claim 1, characterized in that, In S5, the following interactive interfaces are dynamically generated based on the data structure of the business model: intelligent navigation, intelligent Q&A, and intelligent data query. Intelligent navigation: Matches the business function menus or page routes of the business model. When a match is successful, it directly jumps to the corresponding function page, displays relevant data dashboards, or provides quick operation access. Intelligent Q&A: Match business model information, search corporate documents or policies, and extract or answer questions based on user profiles and search results; Intelligent Question Data: Matches business model information with external data sources and business data to perform semantic understanding of question data and output reports.

8. The intelligent form filling method based on multimodal input and dynamic configuration according to claim 1, characterized in that, In S6, the step of performing the submission operation according to the interface configuration in the business model includes: For a front-end and back-end separated model, construct an HTTP request, call the back-end API, and simultaneously upload the attachment stream or pass the attachment reference ID; For front-end and back-end not separated, a hidden form can be constructed or a browser redirect can be performed directly to redirect the user to the traditional system on the other end.

9. An intelligent form-filling system based on multimodal input and dynamic configuration, characterized in that, The system for implementing the method according to any one of claims 1 to 8, the system comprising: The business model configuration center is used to define standardized business models and their key attributes and attribute descriptions, and to build dynamic business models. A multimodal intent recognition engine is used to perform multimodal intent recognition on user input data, obtain text intent and attachment features, perform weighted fusion of the text intent and attachment features to obtain the final target business type and load the corresponding dynamic business model; The Enhanced Context module is used to construct a multimodal enhanced context, including: a user profile layer, a spatiotemporal environment layer, a knowledge enhancement layer, attached structured information, and a structured guidance layer; The large model orchestration service module is used to extract structured data from user text and attachment structured information based on the enhanced context, call the large language model, establish the association between fields and attachments, verify the compliance and completeness of fields and attachments, and when a field or attachment is detected to be missing, the external data adapter actively pulls external data sources to obtain the missing content to supplement it until the field and attachment are complete. The session state manager is used to dynamically generate answer information and forms based on the data structure of the business model, automatically fill the corresponding fields of the form with the extracted and validated data, and display the form in a visual UI interface; it also maintains the form filling status, filled data, attachment list and context history of the current session. The backend adaptation gateway is used to execute the submission operation according to the interface configuration in the business model after receiving the user's confirmation submission instruction. After the submission is completed, the session flow is performed, allowing users to choose to enter a new form scenario or end the session.

10. The intelligent form-filling system based on multimodal input and dynamic configuration according to claim 9, characterized in that, The multimodal intent recognition engine integrates keyword retrieval, vector retrieval, small model classifier, and multimodal attachment parser.

Images