Form content autonomous efficient filling method and device based on multi-modal large model

Abstract

The invention discloses a form content autonomous efficient filling method and device based on a multi-modal large model. The method comprises the following steps: receiving input data in a text, image, document or voice form; performing cross-modal information deep fusion and feature extraction on the input data through the multi-modal large model; a knowledge base and historical data are called, and form filling content is generated through intelligent reasoning and semantic conversion; and performing compliance decision judgment on the generated content, if the generated content is normal, finishing filling and recording a log, and if the generated content is illegal, finishing filling and recording the log after exception processing. Wherein cosine similarity calculation is adopted for cross-modal feature correlation analysis, weighted summation is adopted for feature fusion, corresponding formulas are adopted for semantic understanding result optimization and weight updating, and a scoring formula is adopted for compliance quantification. According to the method, multi-modal input can be efficiently processed, and the accuracy and compliance of form filling are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of chart processing, and in particular to a method and device for autonomously and efficiently filling form content based on a multimodal large model. Background Art

[0002] Currently, in the field of form processing, existing form filling systems have many problems and are unable to meet the requirements of efficient and accurate filling in practical applications.

[0003] On the one hand, existing form filling systems lack intelligence and struggle to effectively handle complex semantics and unstructured input. Among the technical solutions they employ, one is a rule-based engine-driven approach, relying on predefined business rules (such as regular expressions) to match field content. This approach relies too heavily on fixed rules and often fails to accurately understand and extract valid content when faced with complex semantic expressions and unstructured input.

[0004] On the other hand, existing systems lack adaptability and generalization capabilities when dealing with forms in different industries and formats. Some systems integrate with intelligent customer service, allowing users to provide information through natural language interaction, which the system then parses and fills out. However, these systems also rely on predefined rules or templates. When forms involve specialized terminology and special formatting requirements from different industries, these rules or templates need to be frequently adjusted, otherwise they will not function properly.

[0005] Furthermore, existing technologies are unable to integrate multi-source information, such as images, voice, and text. This can lead to information loss or errors during processing, further impacting the accuracy and completeness of form filling and failing to meet the multimodal information processing requirements of practical applications. Due to the aforementioned shortcomings of existing technologies, the efficiency and quality of form filling are limited. Therefore, a method and device for autonomously and efficiently filling form content that can address these issues is urgently needed. Summary of the Invention

[0006] To this end, the present invention provides a method and device for autonomously and efficiently filling form content based on a multimodal large model, which solves the problems of insufficient intelligence in existing form filling, difficulty in dealing with complex semantics and unstructured input; poor adaptability and insufficient generalization ability for forms in different industries and formats; and inability to integrate multi-source information, resulting in information loss or errors.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a method for autonomously and efficiently filling form content based on a multimodal large model, comprising the following steps:

[0008] Receive input data in the form of text, images, documents, or voice;

[0009] Inputting the input data into a multimodal large model, and performing cross-modal information deep fusion and feature extraction on the input data through the multimodal large model;

[0010] The knowledge base and historical data are called, and the multimodal large model performs intelligent reasoning and semantic conversion based on the fused and extracted features, as well as the knowledge base and historical data, to generate form filling content;

[0011] Performing a compliance decision on the generated form filling content; if the compliance decision is normal, completing the form filling and recording a log;

[0012] If the compliance decision is judged as a violation, it will be processed through the exception handling component, and then the form will be completed and the log will be recorded.

[0013] As a preferred solution for the autonomous and efficient form content filling method based on a multimodal large model, the multimodal large model performs deep cross-modal information fusion and feature extraction on the input data, specifically including:

[0014] For different types of input data, corresponding modal features are extracted respectively;

[0015] Perform correlation analysis on the extracted modal features and establish a cross-modal feature mapping relationship. The mapping relationship is calculated using the cosine similarity formula:

[0016]

[0017] Where, F i and F j are the eigenvectors of two different modes respectively;

[0018] Based on the cross-modal feature mapping relationship, the different modal features are fused to form a unified fusion feature representation:

[0019]

[0020] Where, F U The fusion feature representation of different modal features, w i is the weight coefficient of the i-th mode and satisfies

[0021] As a preferred solution for the autonomous and efficient form filling method based on a multimodal large model, the multimodal large model generates form filling content by performing intelligent reasoning and semantic conversion based on fused and extracted features, as well as knowledge base and historical data, including:

[0022] The multimodal large model combines the domain knowledge in the knowledge base to perform semantic understanding of the fusion features;

[0023] Based on the form filling rules and user feedback information recorded in the historical data, the semantic understanding results are optimized and adjusted. The optimization process uses the error correction formula:

[0024] R opt =R ini +α×(TR ini )

[0025] Where R opt To optimize the adjusted semantic understanding results, R ini is the initial semantic understanding result, T is the target result, α is the learning rate and 0<α<1;

[0026] Convert the optimized and adjusted semantic understanding results into fill-in content that meets the form format requirements;

[0027] The historical data includes past form filling data, user feedback on form filling results, and characteristic data of set industries and set format forms. The historical data is used to update the weight using the formula:

[0028]

[0029] Where, is the weight value corresponding to the i-th related factor at the t-th iteration; is the weight value corresponding to the i-th related factor after the t-th iteration; t is the number of iterations, β is the update rate, Δw i is the weight adjustment amount.

[0030] As a preferred solution for the autonomous and efficient form filling method based on a multimodal large model, the compliance decision-making judgment refers to checking the generated form filling content according to preset compliance rules to determine whether it complies with relevant regulations and requirements. The compliance quantification adopts the scoring formula:

[0031]

[0032] Where p k is the weight of the kth compliance rule, C k is the compliance of the kth rule, when the compliance quantitative score S com When the threshold is reached, it is considered compliant.

[0033] As a preferred solution for the autonomous and efficient form content filling method based on a multimodal large model, it is handled through the exception handling component, including:

[0034] Identify and flag illegal content;

[0035] Generate corresponding prompt information and feedback to users based on the violation type to guide them to make corrections;

[0036] Receive the user's corrected information, re-perform intelligent reasoning and semantic conversion, generate new form filling content, and make compliance decisions again.

[0037] The present invention also provides a device for autonomously and efficiently filling form content based on a multimodal large model, comprising:

[0038] A data input module for receiving input data in the form of text, image, document or voice;

[0039] A fusion and extraction module is used to input the input data into a multimodal large model, and perform deep fusion and feature extraction of cross-modal information on the input data through the multimodal large model;

[0040] The reasoning and conversion module is used to call the knowledge base and historical data, and the multimodal large model performs intelligent reasoning and semantic conversion based on the fused and extracted features, as well as the knowledge base and historical data to generate form filling content;

[0041] A compliance decision module is used to make a compliance decision on the generated form filling content; if the compliance decision is normal, the form filling is completed and a log is recorded;

[0042] The exception handling module is used to complete the form filling and record the log if the compliance decision is judged as a violation through the exception handling component.

[0043] As a preferred solution for the autonomous and efficient form content filling device based on a multimodal large model, the fusion extraction module includes:

[0044] A feature extraction unit is used to extract corresponding modal features for different types of input data;

[0045] The correlation analysis unit is used to perform correlation analysis on the extracted modal features and establish a cross-modal feature mapping relationship. The mapping relationship is calculated using the cosine similarity formula:

[0046]

[0047] Where, F i and F j are the eigenvectors of two different modes respectively;

[0048] A feature fusion unit is used to fuse the features of different modalities based on the cross-modal feature mapping relationship to form a unified fusion feature representation:

[0049]

[0050] Where, F U The fusion feature representation of different modal features, wi is the weight coefficient of the i-th mode and satisfies

[0051] As a preferred solution for the autonomous and efficient form content filling device based on a multimodal large model, the reasoning conversion module includes:

[0052] A semantic understanding unit, which is used to combine the multimodal large model with the domain knowledge in the knowledge base to perform semantic understanding on the fusion features;

[0053] The optimization and adjustment unit is used to optimize and adjust the semantic understanding results based on the form filling rules and user feedback information recorded in the historical data. The optimization process uses the error correction formula:

[0054] R opt =R ini +α×(TR ini )

[0055] Where R opt To optimize the adjusted semantic understanding results, R ini is the initial semantic understanding result, T is the target result, α is the learning rate and 0<α<1;

[0056] A result conversion unit is used to convert the optimized and adjusted semantic understanding results into filling content that meets the form format requirements;

[0057] In the inference conversion module, the historical data includes past form filling data, user feedback data on form filling results, and characteristic data of the set industry and set format form. The historical data is used to update the weight using the formula:

[0058]

[0059] Where, is the weight value corresponding to the i-th related factor at the t-th iteration; is the weight value corresponding to the i-th related factor after the t-th iteration; t is the number of iterations, β is the update rate, Δw i is the weight adjustment amount.

[0060] As a preferred solution for the autonomous and efficient form filling device based on a multimodal large model, in the compliance decision module, the compliance decision judgment refers to checking the generated form filling content according to the preset compliance rules to determine whether it complies with relevant regulations and requirements. The compliance quantification adopts the scoring formula:

[0061]

[0062] Where p k is the weight of the kth compliance rule, Ck is the compliance of the kth rule, when the compliance quantitative score S com When the threshold is reached, it is considered compliant.

[0063] As a preferred solution for the autonomous and efficient form content filling device based on a multimodal large model, the exception handling module includes:

[0064] Identification and marking unit, used to identify and mark illegal content;

[0065] The feedback correction unit is used to generate corresponding prompt information and feedback to the user according to the violation type, guiding the user to make corrections;

[0066] The compliance re-judgment unit is used to receive the user's revised information, re-perform intelligent reasoning and semantic conversion, generate new form filling content, and make compliance decision judgments again.

[0067] The present invention has the following advantages:

[0068] First, by using a large multimodal model to deeply integrate and extract features from cross-modal information such as images, text, and voice, it can accurately understand complex semantics and process unstructured inputs, solving the problem that existing systems rely on predefined rules or templates and have difficulty dealing with complex situations.

[0069] Second, the system's adaptability and generalization capabilities are greatly enhanced. By combining historical data and user feedback to dynamically optimize filling strategies, it can adapt to forms in different industries and formats without frequently adjusting rules or templates, overcoming the shortcomings of insufficient generalization capabilities of existing technologies.

[0070] Third, it effectively reduces information loss or errors, supports multi-source information fusion, avoids information processing problems caused by the inability of existing technologies to fuse multimodal information, and improves the accuracy of form filling.

[0071] Fourth, we aim to improve the efficiency of form filling. By leveraging the intelligent reasoning and semantic transformation of knowledge bases and multimodal large models, we can automate information entry and enhance processing efficiency. Through compliance decision-making and exception handling mechanisms, we can further ensure the standardization of form filling content, reduce the risk of errors, and improve overall processing quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and those skilled in the art can, without inventive effort, derive other implementation drawings based on the provided drawings.

[0073] The structures, proportions, sizes, etc. illustrated in this specification are intended solely to complement the contents disclosed herein and to facilitate understanding and reading by persons skilled in the art. They are not intended to limit the conditions under which the present invention may be implemented and therefore have no substantive technical significance. Any structural modifications, changes in proportions, or adjustments in sizes, without affecting the efficacy and objectives of the present invention, shall remain within the scope of the technical contents disclosed herein.

[0074] Figure 1 This is a flow chart of the method for autonomously and efficiently filling in form content based on a multimodal large model provided in Example 1 of the present invention;

[0075] Figure 2 This is a technical flowchart of the method for autonomously and efficiently filling form content based on a multimodal large model provided in Example 1 of the present invention;

[0076] Figure 3 Schematic diagram of the architecture of the device for autonomously and efficiently filling in form content based on a multimodal large model provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0077] The following describes the implementation of the present invention using specific embodiments. Those skilled in the art will readily understand the other advantages and benefits of the present invention from the disclosure herein. Obviously, the embodiments described are only a portion of the present invention, not all of it. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0078] Example 1

[0079] See also Figure 1 and Figure 2 The embodiment of the present invention provides a method for autonomously and efficiently filling form content based on a multimodal large model, comprising the following steps:

[0080] S1. Receive input data in the form of text, image, document or voice;

[0081] Specifically, step S1 is the data input link, which supports multiple modal inputs. This is because in actual applications, users may provide information in different forms, such as uploading ID card pictures (images), entering personal profile text (text), or speaking information (voice), etc. Diversified input methods can improve user convenience.

[0082] S2. Inputting the input data into a large multimodal model, and performing cross-modal information deep fusion and feature extraction on the input data through the large multimodal model;

[0083] Specifically, step S2 uses a multimodal large model to process cross-modal information. The core principle is that the large model has the ability to simultaneously understand and process different types of data, breaking the limitations of a single modality, extracting key features from multiple inputs and fusing them, laying the foundation for subsequent semantic understanding and reasoning.

[0084] S3. Calling the knowledge base and historical data, the multimodal large model performs intelligent reasoning and semantic conversion based on the fused and extracted features, as well as the knowledge base and historical data, to generate form filling content;

[0085] Specifically, the S3 step uses the knowledge base and historical data to assist the large model in reasoning and conversion. The knowledge base provides professional knowledge in various fields, and the historical data contains past filling experience and user feedback. The combination of the two allows the large model to generate form content that better meets actual needs and achieve accurate mapping from input information to form fields.

[0086] S4. Perform a compliance decision on the generated form filling content; if the compliance decision is normal, complete the form filling and record the log;

[0087] S5. If the compliance decision is a violation, the exception handling component will be used to process the violation, complete the form, and record the log.

[0088] Specifically, steps S4 and S5 are compliance control links, which ensure that the filled content complies with relevant regulations through compliance decision-making. If it is normal, the filling is completed and recorded directly. If it is a violation, it will be completed after exception handling. This process can ensure the legality and accuracy of the form information, and log records facilitate subsequent tracing and analysis.

[0089] In this embodiment, in step S2, the multimodal large model performs deep cross-modal information fusion and feature extraction on the input data, specifically including: extracting corresponding modal features for different types of input data; performing correlation analysis on the extracted modal features, and establishing a cross-modal feature mapping relationship, which is calculated using the cosine similarity formula:

[0090]

[0091] Where, F i and F j are feature vectors of two different modalities respectively; based on the cross-modal feature mapping relationship, the different modal features are fused to form a unified fusion feature representation:

[0092]

[0093] Where, F U The fusion feature representation of different modal features, w iis the weight coefficient of the i-th mode and satisfies

[0094] Specifically, corresponding modal features are extracted for different types of input data because the information representation of different modal data (such as text and images) varies greatly. Text is characterized by character sequences, and images are characterized by pixel matrices. Separate extraction can retain the unique information of each modality.

[0095] The cosine similarity formula is used to calculate cross-modal feature mapping relationships. The principle is that cosine similarity measures the directional similarity between two vectors. The closer the value is to 1, the higher the correlation between the two modal features. For example, the feature vector of the description of "red" in text and the red area in the image can be calculated with a high degree of similarity using this formula, thus establishing an effective mapping relationship and determining whether different modal information refers to the same content.

[0096] The fusion feature representation formula uses weighted coefficients to sum the features of each modality. The weight coefficients reflect the importance of different modalities in the current form filling task. For example, when filling in academic information, the weight of the academic certificate image (image modality) may be higher than the oral voice information (voice modality). By setting the weights reasonably and ensuring that the sum is 1, multiple modal features can be integrated into a unified feature vector F. U , which facilitates unified processing and understanding of subsequent large models.

[0097] In this embodiment, in step S3, the multimodal large model performs intelligent reasoning and semantic conversion based on the fused and extracted features, as well as the knowledge base and historical data, to generate form filling content, including: the multimodal large model combines the domain knowledge in the knowledge base to perform semantic understanding on the fused features; based on the form filling rules recorded in the historical data and user feedback information, the semantic understanding results are optimized and adjusted. The optimization process uses the error correction formula:

[0098] R opt =R ini +α×(TR ini )

[0099] Where R opt To optimize the adjusted semantic understanding results, R ini is the initial semantic understanding result, T is the target result, α is the learning rate and 0<α<1;

[0100] Convert the optimized and adjusted semantic understanding results into fill-in content that meets the form format requirements;

[0101] Specifically, semantic understanding is achieved by combining domain knowledge from a knowledge base. The knowledge base stores a wealth of specialized terminology, industry rules, and other information, helping the large model accurately understand the specialized semantics of fused features. For example, when filling out medical forms, the medical terminology in the knowledge base allows the large model to correctly understand the meaning of specialized descriptions such as "CT images show shadows in the lungs."

[0102] Among them, the error correction formula is used to optimize the semantic understanding results, and the initial result R ini There may be deviations. The target result T is the ideal result determined based on historical data and knowledge base. The learning rate α controls the amplitude of each optimization to avoid excessive or insufficient adjustments. Through this formula, the optimized result R opt Gradually approach the goal and improve the accuracy of semantic understanding.

[0103] The optimized results are converted into form format content because the form has fixed field format requirements, and the natural language information after semantic understanding needs to be converted into content that conforms to the field specifications (such as the date format must be "year-month-day").

[0104] In a possible embodiment, the historical data includes past form filling data, user feedback data on form filling results, and characteristic data of set industries and set format forms. The historical data is used to update the weight using the formula:

[0105]

[0106] Where, is the weight value corresponding to the i-th related factor at the t-th iteration; is the weight value corresponding to the i-th related factor after the t-th iteration; t is the number of iterations, β is the update rate, Δw i is the weight adjustment amount.

[0107] Specifically, the weight update formula of historical data reflects the system's adaptability, and dynamic adjustment is achieved through the number of iterations t. Weight adjustment amount Δw i Based on historical data calculation, it reflects the changes in the importance of different factors (such as different modal inputs) in actual filling. The update rate β controls the adjustment amplitude so that the weight w i It can be continuously optimized as experience accumulates, allowing the system to better adapt to different scenarios.

[0108] In this embodiment, in step S4, the compliance decision judgment refers to checking the generated form filling content according to the preset compliance rules to determine whether it complies with relevant regulations and requirements. The compliance quantification adopts the scoring formula:

[0109]

[0110] Where p k is the weight of the kth compliance rule, C k is the compliance of the kth rule, when the compliance quantitative score S com When the threshold is reached, it is considered compliant.

[0111] Specifically, compliance decision judgment determines whether the filled content is compliant through quantitative scoring. The preset compliance rules cover the requirements of laws and regulations, industry standards, etc. (such as personal information protection rules, financial data format rules). The weight p of the kth rule k Indicates the importance of the rule, compliance C k 1 means compliance, 0 means non-compliance. The total score S is obtained by weighted summation. com , compared with the threshold. If it is higher than or equal to the threshold, it is compliant; otherwise, it is non-compliant. This quantitative method makes compliance judgment more objective and accurate, and can effectively avoid the subjectivity and omissions of manual judgment.

[0112] In this embodiment, in step S5, processing is performed through the exception handling component, including: identifying and marking the illegal content; generating corresponding prompt information feedback to the user based on the violation type, guiding the user to make corrections; receiving the user's corrected information, re-performing intelligent reasoning and semantic conversion, generating new form filling content, and making compliance decision judgments again.

[0113] Specifically, the role of the exception handling component is to correct illegal content to ensure the legality of the form. First, identify and mark illegal content to clarify the problem (such as the ID card number format is incorrect, the amount filled in is out of range); then generate prompts based on the violation type, which can accurately guide users to make corrections and improve correction efficiency (such as the prompt "ID card number should be 18 digits, please re-enter"); after receiving the corrected information, reprocess and re-judge compliance to form a closed loop, ensuring that the final filled content fully meets the requirements and guarantees the accuracy and compliance of the form information.

[0114] The application scenarios of the present invention are as follows:

[0115] Scenario 1: Financial Services Scenario:

[0116] When applying for loans, insurance claims, and processing credit cards, users are required to submit multimodal information, including ID cards, proof of income (images), bank statements (documents), professional information (text), and oral descriptions of their assets (voice). This method uses a large multimodal model to deeply fuse and extract features from this cross-modal information. By combining financial domain knowledge bases (such as loan qualification review standards and insurance clauses) with historical fill-in data (such as form filling patterns of similar users), it generates fill-in content that conforms to the form format. Compliance judgments are then made based on financial regulatory compliance rules (such as personal information protection and data format specifications), ensuring that form information is accurate and complies with industry regulations, significantly improving the efficiency and standardization of financial business form processing.

[0117] Scenario 2: Medical and health scenario:

[0118] For scenarios such as electronic medical record entry, examination application form completion, and medical insurance reimbursement form processing, doctors or patients may provide information such as medical record text, CT / MRI images, diagnostic voice recordings, and past medical documentation. The multimodal large model in this method integrates this information, combines it with a medical knowledge base (such as disease terminology and treatment protocols) to understand semantics, and generates standardized form content (such as symptom descriptions, examination items, and diagnosis results). Compliance checks are also performed based on medical industry compliance rules (such as medical record writing standards and medical insurance reimbursement policies). Prompts are provided to guide corrections for abnormal content, ensuring the professionalism and compliance of medical form information and reducing the paperwork burden on medical staff.

[0119] Scenario 3: Government Service Scenario:

[0120] In government services such as social security registration, provident fund withdrawal, and business license application, users often need to submit various types of supporting documents (such as household registration books, images of property certificates), personal information forms (documents), and business description text or voice. This method can autonomously receive and process these multimodal inputs, calling on government domain knowledge bases (such as service guidelines and policy clauses) and historical processing data (such as form filling features for similar services) to generate fill-in content. It then uses preset government compliance rules (such as material format requirements and information authenticity verification standards) to make judgments, ensuring that the form complies with government processing regulations, reducing user on-site queuing and manual data entry time, and improving government service efficiency.

[0121] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied in a distributed scenario and completed by multiple devices working together. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method for autonomously and efficiently filling form content based on a multimodal large model.

[0122] It should be noted that the above description is limited to some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that described in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0123] Example 2

[0124] See also Figure 3 Embodiment 2 of the present invention further provides a device for autonomously and efficiently filling form content based on a multimodal large model, comprising:

[0125] The data input module 100 is used to receive input data in the form of text, image, document or voice;

[0126] A fusion and extraction module 200 is used to input the input data into a multimodal large model, and perform cross-modal information deep fusion and feature extraction on the input data through the multimodal large model;

[0127] The reasoning and conversion module 300 is used to call the knowledge base and historical data, and the multimodal large model performs intelligent reasoning and semantic conversion based on the fused and extracted features, as well as the knowledge base and historical data, to generate form filling content;

[0128] The compliance decision module 400 is used to make a compliance decision on the generated form filling content; if the compliance decision is normal, the form filling is completed and a log is recorded;

[0129] The exception handling module 500 is used to complete the form filling and record the log after processing it through the exception handling component if the compliance decision is judged to be a violation.

[0130] In this embodiment, the fusion extraction module 200 includes:

[0131] The feature extraction unit 201 is used to extract corresponding modal features for different types of input data;

[0132] The association analysis unit 202 is used to perform association analysis on the extracted modal features and establish a cross-modal feature mapping relationship. The mapping relationship is calculated using the cosine similarity formula:

[0133]

[0134] Where, F i and F jare the eigenvectors of two different modes respectively;

[0135] The feature fusion unit 203 is used to fuse the features of different modalities based on the cross-modal feature mapping relationship to form a unified fusion feature representation:

[0136]

[0137] Where, F U The fusion feature representation of different modal features, w i is the weight coefficient of the i-th mode and satisfies

[0138] In this embodiment, the reasoning conversion module 300 includes:

[0139] Semantic understanding unit 301, used for combining the multimodal large model with the domain knowledge in the knowledge base to perform semantic understanding on the fusion features;

[0140] The optimization and adjustment unit 302 is used to optimize and adjust the semantic understanding results based on the form filling rules and user feedback information recorded in the historical data. The optimization process uses the error correction formula:

[0141] R opt =R ini +α×(TR ini )

[0142] Where R opt To optimize the adjusted semantic understanding results, R ini is the initial semantic understanding result, T is the target result, α is the learning rate and 0<α<1;

[0143] A result conversion unit 303 is used to convert the optimized and adjusted semantic understanding result into filling content that meets the form format requirements;

[0144] In the inference conversion module 300, the historical data includes past form filling data, user feedback data on form filling results, and characteristic data of the set industry and set format form. The historical data is used to update the weight using the formula:

[0145]

[0146] Where, is the weight value corresponding to the i-th related factor at the t-th iteration; is the weight value corresponding to the i-th related factor after the t-th iteration; t is the number of iterations, β is the update rate, Δw i is the weight adjustment amount.

[0147] In this embodiment, in the compliance decision module 400, the compliance decision judgment refers to checking the generated form filling content according to the preset compliance rules to determine whether it complies with relevant regulations and requirements. The compliance quantification adopts the scoring formula:

[0148]

[0149] Where p k is the weight of the kth compliance rule, C k is the compliance of the kth rule, when the compliance quantitative score S com When the threshold is reached, it is considered compliant.

[0150] In this embodiment, the exception handling module 500 includes:

[0151] Identification and marking unit 501, used to identify and mark illegal content;

[0152] Feedback correction unit 502, used to generate corresponding prompt information based on the violation type and feedback it to the user to guide the user to make corrections;

[0153] The compliance re-judgment unit 503 is used to receive the user's revised information, re-perform intelligent reasoning and semantic conversion, generate new form filling content, and make compliance decision judgment again.

[0154] It should be noted that the information interaction, execution process, etc. between the modules of the above-mentioned device are based on the same concept as the method embodiment in Example 1 of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown above in the present application, and will not be repeated here.

[0155] Example 3

[0156] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium, in which program code for a method for autonomously and efficiently filling in form content based on a multimodal large model is stored. The program code includes instructions for executing embodiment 1 or any possible implementation thereof.

[0157] Computer-readable storage media can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0158] Example 4

[0159] Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor;

[0160] The processor and the memory communicate with each other via a bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the autonomous and efficient form content filling method based on a multimodal large model of Example 1 or any possible implementation thereof.

[0161] Specifically, the processor can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor, which is implemented by reading software code stored in a memory. The memory can be integrated into the processor or located outside the processor and exist independently.

[0162] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode.

[0163] Obviously, those skilled in the art will appreciate that the various modules or steps of the present invention described above can be implemented using a general-purpose computing device, centralized on a single computing device, or distributed across a network of multiple computing devices. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. In some cases, the steps shown or described can be performed in a different order than that shown, or can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0164] Although the present invention has been described in detail above using general descriptions and specific embodiments, it will be apparent to those skilled in the art that modifications and improvements may be made thereto. Therefore, such modifications and improvements, without departing from the spirit of the present invention, are intended to be within the scope of protection claimed herein.

Claims

1. A method for autonomous and efficient filling of form content based on a multimodal large model, characterized by: The following steps are involved: Receive input data in the form of text, images, documents, or voice; Inputting the input data into a multimodal large model, and performing cross-modal information deep fusion and feature extraction on the input data through the multimodal large model; The knowledge base and historical data are called, and the multimodal large model performs intelligent reasoning and semantic conversion based on the fused and extracted features, as well as the knowledge base and historical data, to generate form filling content; Making compliance decisions on the generated form-filled content; If the compliance decision is normal, complete the form and record the log; If the compliance decision is judged as a violation, it will be processed through the exception handling component, and then the form will be completed and the log will be recorded.

2. The method for autonomously and efficiently filling form content based on a multimodal large model according to claim 1 is characterized in that: The multimodal large model performs deep cross-modal information fusion and feature extraction on the input data, specifically including: For different types of input data, corresponding modal features are extracted respectively; Perform correlation analysis on the extracted modal features and establish a cross-modal feature mapping relationship. The mapping relationship is calculated using the cosine similarity formula: Where, F i and F j are the eigenvectors of two different modes respectively; Based on the cross-modal feature mapping relationship, the different modal features are fused to form a unified fusion feature representation: Where, F U The fusion feature representation of different modal features, w i is the weight coefficient of the i-th mode and satisfies 3. The method for autonomously and efficiently filling form content based on a multimodal large model according to claim 1 is characterized in that: The multimodal large model performs intelligent reasoning and semantic conversion based on the fused and extracted features, knowledge base, and historical data to generate form filling content, including: The multimodal large model combines the domain knowledge in the knowledge base to perform semantic understanding of the fusion features; Based on the form filling rules and user feedback information recorded in the historical data, the semantic understanding results are optimized and adjusted. The optimization process uses the error correction formula: R opt =R ini +α×(T-R ini ) Where R opt To optimize the adjusted semantic understanding results, R ini is the initial semantic understanding result, T is the target result, α is the learning rate and 0<α<1; Convert the optimized and adjusted semantic understanding results into fill-in content that meets the form format requirements; The historical data includes past form filling data, user feedback on form filling results, and characteristic data of set industries and set format forms. The historical data is used to update the weight using the formula: Where, is the weight value corresponding to the i-th related factor at the t-th iteration; is the weight value corresponding to the i-th related factor after the t-th iteration; t is the number of iterations, β is the update rate, Δw i is the weight adjustment amount.

4. The method for autonomously and efficiently filling form content based on a multimodal large model according to claim 1 is characterized in that: The compliance decision-making judgment refers to checking the generated form filling content according to the preset compliance rules to determine whether it complies with relevant regulations and requirements. The compliance quantification adopts the scoring formula: Where p k is the weight of the kth compliance rule, C k is the compliance of the kth rule, when the compliance quantitative score S com When the threshold is reached, it is considered compliant.

5. The method for autonomously and efficiently filling form content based on a multimodal large model according to claim 1 is characterized in that: Handled by exception handling components, including: Identify and flag illegal content; Generate corresponding prompt information and feedback to users based on the violation type to guide them to make corrections; Receive the user's corrected information, re-perform intelligent reasoning and semantic conversion, generate new form filling content, and make compliance decisions again.

6. A device for autonomously and efficiently filling in form content based on a multimodal large model, characterized in that: include: A data input module for receiving input data in the form of text, image, document or voice; A fusion and extraction module is used to input the input data into a multimodal large model, and perform deep fusion and feature extraction of cross-modal information on the input data through the multimodal large model; The reasoning and conversion module is used to call the knowledge base and historical data, and the multimodal large model performs intelligent reasoning and semantic conversion based on the fused and extracted features, as well as the knowledge base and historical data to generate form filling content; A compliance decision module is used to make a compliance decision on the generated form filling content; if the compliance decision is normal, the form filling is completed and a log is recorded; The exception handling module is used to complete the form filling and record the log if the compliance decision is judged as a violation through the exception handling component.

7. The device for autonomously and efficiently filling form contents based on a multimodal large model according to claim 6 is characterized in that: The fusion extraction module includes: A feature extraction unit is used to extract corresponding modal features for different types of input data; The correlation analysis unit is used to perform correlation analysis on the extracted modal features and establish a cross-modal feature mapping relationship. The mapping relationship is calculated using the cosine similarity formula: Where, F i and F j are the eigenvectors of two different modes respectively; A feature fusion unit is used to fuse the features of different modalities based on the cross-modal feature mapping relationship to form a unified fusion feature representation: Where, F U The fusion feature representation of different modal features, w i is the weight coefficient of the i-th mode and satisfies 8. The device for autonomously and efficiently filling form contents based on a multimodal large model according to claim 6 is characterized in that: The reasoning conversion module includes: A semantic understanding unit, which is used to combine the multimodal large model with the domain knowledge in the knowledge base to perform semantic understanding on the fusion features; The optimization and adjustment unit is used to optimize and adjust the semantic understanding results based on the form filling rules and user feedback information recorded in the historical data. The optimization process uses the error correction formula: R opt =R ini +α×(T-R ini ) Where R opt To optimize the adjusted semantic understanding results, R ini is the initial semantic understanding result, T is the target result, α is the learning rate and 0<α<1; A result conversion unit is used to convert the optimized and adjusted semantic understanding results into filling content that meets the form format requirements; In the inference conversion module, the historical data includes past form filling data, user feedback data on form filling results, and characteristic data of the set industry and set format form. The historical data is used to update the weight using the formula: Where, is the weight value corresponding to the i-th related factor at the t-th iteration; is the weight value corresponding to the i-th related factor after the t-th iteration; t is the number of iterations, β is the update rate, Δw i is the weight adjustment amount.

9. The device for autonomously and efficiently filling form contents based on a multimodal large model according to claim 6 is characterized in that: In the compliance decision module, the compliance decision judgment refers to checking the generated form filling content according to the preset compliance rules to determine whether it complies with relevant regulations and requirements. The compliance quantification adopts the scoring formula: Where p k is the weight of the kth compliance rule, C k is the compliance of the kth rule, when the compliance quantitative score S com When the threshold is reached, it is considered compliant.

10. The device for autonomously and efficiently filling form contents based on a multimodal large model according to claim 6 is characterized in that: The exception handling module includes: Identification and marking unit, used to identify and mark illegal content; The feedback correction unit is used to generate corresponding prompt information and feedback to the user according to the violation type, guiding the user to make corrections; The compliance re-judgment unit is used to receive the user's revised information, re-perform intelligent reasoning and semantic conversion, generate new form filling content, and make compliance decision judgments again.

Images