Document identification and checking method and device based on large model

Abstract

The invention provides a document recognition and checking method and device based on a large model. The method comprises the steps that a document to be recognized is obtained, the document is converted into a picture format, the obtained document picture is input into a pre-trained visual large model, and recognized information is output. Inputting information recognized by the visual large model into a language large model, writing field information needing to be extracted through a cue word project, guiding the language large model to perform keyword extraction on the input information, and outputting the field information needing to be extracted in a structured mode; configuring a checking rule, including constructing a checking script and a checking rule knowledge base; inputting the output data into a large checking model, constructing a dynamic double-engine retrieval mechanism by the large checking model in combination with an RAG retrieval enhancement generation technology, and checking the input information; and generating a checking result, and performing visual display. According to the invention, the accuracy and robustness of recognition of various documents can be improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of natural language processing, and in particular relates to a document recognition and checking method and device based on a large model. BACKGROUND

[0002] In the daily operation and project promotion process of various industries, a large number of documents with complex structure and strong professional nature are generated and used. The traditional document processing method mainly relies on manual recognition and checking, which requires professional personnel to review page by page. This method not only has low efficiency, but is also easily affected by subjective factors, and is difficult to meet the stringent requirements of various industries for efficient and accurate document processing.

[0003] However, in the current document processing field, the traditional document recognition and checking method mainly relies on optical character recognition (OCR) technology and preset rule checking algorithm. Although the OCR technology can extract the text in the document from the picture, it has low recognition accuracy in the face of complex layout, blurred handwriting, and handwritten text, and is prone to problems such as missed recognition and misrecognition. The preset rule checking algorithm needs to manually write a large number of rules for different types of documents and checking requirements, which is not only time-consuming and labor-intensive, but also lacks flexibility and universality, and is difficult to cope with the diversification of document formats and contents.

[0004] With the rapid development of large model technology, its powerful language understanding and reasoning ability provides new possibilities for document processing. However, there is currently no mature solution to effectively apply large model technology to the complete process of document recognition and checking, and truly realize efficient, accurate and adaptive document processing. This also means that how to fully tap the potential of large model technology in the field of document processing is still a key problem to be solved. SUMMARY

[0005] Therefore, the present application aims to overcome the above-mentioned problems in the prior art, and proposes a document recognition and checking method and device based on a large model, which utilizes the powerful capabilities of large models to improve the accuracy and robustness of various document recognition, while realizing automatic and intelligent document checking, reducing the cost of manual intervention, and improving the efficiency and quality of document processing.

[0006] To achieve the above-mentioned purposes, the technical solution of the present application is as follows:

[0007] In a first aspect, the present application provides a document recognition and checking method based on a large model, comprising the following steps:

[0008] Step 1: Obtain the document to be recognized, and convert the document into a picture format;

[0009] Step 2: input the obtained document image into a pre-trained visual large model, and output the recognized information;

[0010] Step 3: input the information recognized by the visual large model into a language large model, write the required field information through prompt word engineering, guide the language large model to extract keywords from the input information, and output the required field information in a structured manner;

[0011] Step 4: configure review rules, including building review scripts and review rule knowledge base;

[0012] Step 5: input the data output by steps 3 and 4 into a review large model, and the review large model combines RAG retrieval enhancement generation technology to build a dynamic double-engine retrieval mechanism to review the input data;

[0013] Step 6: generate review results and visualize them.

[0014] Further, step 2 specifically includes:

[0015] Collect document image data sets, and convert multiple types of document image data to images, remove skewed and blurred document image data;

[0016] Annotate the collected document images, and perform polygon annotation on text areas, table cells, and formula block elements. Also, annotate the field information to be extracted. Build the annotation results into a data set, and generate training set, validation set, and test set through custom scripts;

[0017] Pass the annotated document image data into the visual large model and adjust the loss function;

[0018] Fuse the recognition results in multiple modalities to output the recognition results in the form of structured data, including picture text content, text position, paragraph structure, table information, and corresponding recognition box coordinate information.

[0019] Further, step 3 includes:

[0020] Input the information recognized by the visual large model into the language large model;

[0021] Write targeted prompts according to business needs;

[0022] The language large model reasons according to the prompts, extracts the corresponding key information from the information recognized by the visual large model, and uses the attention mechanism and pre-trained semantic understanding ability to extract the required data field;

[0023] Structure the extracted key information according to the preset format and output standardized data.

[0024] Further, step 4 includes:

[0025] Store general check rules and industry-specific specifications in the check rule knowledge base;

[0026] Configure a professional rule engine as the core carrier of rule execution and scheduling;

[0027] Design prompt word templates through natural language, including four modules of task description, knowledge reference, constraint condition and output format, to clearly define the checking target and requirements.

[0028] Further, step 4 also includes:

[0029] Based on business feedback or reinforcement learning algorithm, real-time adjustment of prompt word semantic expression and parameter setting.

[0030] Further, in step 5, the check large model combines RAG retrieval enhancement generation technology to build a dynamic double-engine retrieval mechanism, and the checking of the input information includes:

[0031] Calculate the semantic similarity of structured data, and recall the highest matching basic check rule from the check rule knowledge base;

[0032] Through keyword extraction and rule template matching, trigger rules to filter business-specific verification logic from the dynamic rule library;

[0033] In the model inference stage, the check large model fuses business system data, recalled basic check rules, business-specific verification logic and data to be checked for inference, and performs deep checking from multiple dimensions.

[0034] In a second aspect, the present application provides a document recognition and checking device based on a large model, comprising:

[0035] A data acquisition device for acquiring a document to be identified and converting the document into a picture format;

[0036] A first processing device for inputting the obtained document picture into a pre-trained visual large model and outputting the recognized information;

[0037] A second processing device for inputting the information recognized by the visual large model into a language large model, writing the field information to be extracted through prompt word engineering, guiding the language large model to extract keywords from the input information, and outputting the field information to be extracted in a structured manner;

[0038] A third processing device for configuring check rules, including building check scripts and a check rule knowledge base;

[0039] The fourth processing device is configured to input the data output by the second processing device and the third processing device into a verification large model, and the verification large model is configured to perform verification on the input data by combining a RAG retrieval enhancement technology to construct a dynamic double-engine retrieval mechanism.

[0040] The fifth processing device is configured to generate a verification result and perform visual display.

[0041] In a third aspect, the present application provides an electronic device, which comprises a processor and a memory connected with the processor and configured to store executable instructions of the processor, and the processor is configured to execute the above-mentioned large model-based document identification and verification method.

[0042] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is configured to execute the above-mentioned large model-based document identification and verification method when executed by a processor.

[0043] Compared with the prior art, the large model-based document identification and verification method and device have the following advantages:

[0044] The present application innovatively combines visual large models (document image understanding) and language large models (semantic analysis) to realize end-to-end conversion from unstructured documents to structured data. It breaks through the limitation of traditional OCR that can only identify text content and simultaneously analyzes complex features such as layout structure, table logical relationship, and paragraph semantic association.

[0045] In the large model verification stage, the present application constructs a dynamic double-engine retrieval mechanism. On the one hand, it calculates the semantic similarity to recall the highest matching degree of the basic verification rules, and on the other hand, it extracts keywords and matches the business-specific verification logic through the rule template. This mechanism can flexibly combine general rules and business-specific rules to improve the accuracy and adaptability of verification. It can quickly adjust the verification logic according to different business scenarios, avoiding the limitations of a single rule base, reducing manual intervention, and improving verification efficiency.

[0046] In the verification process, the present application introduces RAG retrieval enhancement technology. For example, in the contract verification scenario, the latest provisions are extracted from the regulation library to combine with the contract provisions for compliance comparison, and in the financial statement processing, historical data is called to verify the accuracy of the data logic of the accounting standards. By obtaining external authoritative information in real time, the basis and reliability of the verification are enhanced, and the verification result is ensured to comply with the latest regulations and industry standards.

[0047] The present application stores general verification rules and industry-specific specifications (such as legal provisions and industry standards) in a knowledge base, realizes dynamic updating and expansion of verification rules, ensures that the verification process always complies with the latest industry standards and regulations, improves the guarantee of document compliance, and reduces legal risks.

[0048] The application closely combines the recognition process and the checking process of the document, realizes the integrated operation of recognition and checking. The user does not need to switch between multiple steps, and can complete the recognition and checking of the document in one process, which significantly improves the work efficiency and reduces the errors and delays in information transmission.

[0049] The application designs a prompt word strategy to guide the checking large model to generate a checking result. The generated checking result not only includes prompt information of whether the document content conforms to the checking rule, but also specifically points out the error content, error type, error position when there is a situation that does not conform to the checking rule, and provides corresponding modification suggestions. At the same time, the checking result is displayed to the user in a visual form, and the user is prompted through a pop-up window, so that the user can quickly understand the problems existing in the document and make modifications. BRIEF DESCRIPTION OF DRAWINGS

[0050] The accompanying drawings, which form a part of the present application, are included to provide a further understanding of the application, and are incorporated herein for purposes of illustrating the preferred embodiments of the present application and explaining the principles of the application. In the drawings:

[0051] Figure 1 The accompanying drawings, which form a part of the present application, are included to provide a further understanding of the application, and are incorporated herein for purposes of illustrating the preferred embodiments of the present application and explaining the principles of the application. In the drawings:

[0052] Figure 2 The accompanying drawings, which form a part of the present application, are included to provide a further understanding of the application, and are incorporated herein for purposes of illustrating the preferred embodiments of the present application and explaining the principles of the application. In the drawings:

[0053] Figure 3 The accompanying drawings, which form a part of the present application, are included to provide a further understanding of the application, and are incorporated herein for purposes of illustrating the preferred embodiments of the present application and explaining the principles of the application. In the drawings:

[0054] Figure 4 The accompanying drawings, which form a part of the present application, are included to provide a further understanding of the application, and are incorporated herein for purposes of illustrating the preferred embodiments of the present application and explaining the principles of the application. In the drawings:

[0055] Figure 5 The accompanying drawings, which form a part of the present application, are included to provide a further understanding of the application, and are incorporated herein for purposes of illustrating the preferred embodiments of the present application and explaining the principles of the application. In the drawings:

[0056] Figure 6 The accompanying drawings, which form a part of the present application, are included to provide a further understanding of the application, and are incorporated herein for purposes of illustrating the preferred embodiments of the present application and explaining the principles of the application. In the drawings:

[0057] Figure 7 The accompanying drawings, which form a part of the present application, are included to provide a further understanding of the application, and are incorporated herein for purposes of illustrating the preferred embodiments of the present application and explaining the principles of the application. In the drawings:

[0058] Figure 8 The accompanying drawings, which form a part of the present application, are included to provide a further understanding of the application, and are incorporated herein for purposes of illustrating the preferred embodiments of the present application and explaining the principles of the application. In the drawings: DETAILED DESCRIPTION

[0059] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0060] The present application will be described in detail below with reference to the accompanying drawings and embodiments.

[0061] Embodiment 1:

[0062] As Figure 1 shown, the present application provides a large model-based document recognition and checking method, the specific steps are as follows:

[0063] 1. Document picture acquisition: Obtain the picture of paper document through scanning equipment, photographing equipment, etc., or directly obtain the picture format file of electronic document as the original data to be processed.

[0064] 2. Visual large model identifies pictures: input the obtained document picture into the pre-trained visual large model, the visual large model is trained by a large amount of document image data and has the recognition ability of various document layouts, characters and symbols. The visual large model analyzes the document picture, identifies the text content, text position, paragraph structure, table information, etc. in the picture, supports multi-element recognition, and finally outputs the recognition result in the form of structured data. For example, for a document picture containing a table, the visual large model can identify the row and column structure of the table and the cell content, and convert them into editable table data; for paragraph text, it can accurately identify the text order and paragraph hierarchy, analyze the logical relationship between sentences in the paragraph, such as cause and effect relationship, parallel relationship, progressive relationship, etc. By identifying logical conjunctions (such as “therefore”, “but”, “first”, “second”, etc.) and contextual semantics, the logical structure of the paragraph is constructed. Finally, the recognized title, text, handwritten characters, table, seal, date and recognition box information, etc. are output. The visual large model identification process is shown in Figure 2 , which specifically includes:

[0065] (1) Collect real document image data set, convert multiple types of data into images, remove skewed and blurred data. Apply Laplacian operator to the image , calculate the variance of the image to evaluate the blur degree:

[0066]

[0067] wherein, is the total number of image pixels, is the average value of image pixels after Laplace transform. When is lower than a certain threshold , the image is determined to be blurred and removed.

[0068] (2) Label the collected document images, and perform polygon labeling on text regions, table cells, formula blocks, and other elements. Finally, construct the labeling results into a dataset, and generate training / validation / test sets (ratio 8:1:1) through a custom script. The purpose is to effectively evaluate the model performance when training the model, and prevent overfitting. The specific division formula is as follows:

[0069]

[0070] wherein, is the total data volume of the screened images.

[0071] (3) The labeled document image data is transmitted into a visual large model, and the loss function is fine-tuned. The specific formula is as follows:

[0072]

[0073] wherein, θ is a model parameter, f(x i ; θ) is a model output, is a true label. LoRA technology is used for lightweight fine-tuning to reduce training cost. A low-rank matrix is referenced to reduce training cost. The LoRA fine-tuning formula is as follows:

[0074]

[0075] (4) The recognition results are fused in multiple modalities, and output in the form of structured data. The output content includes recognized title content, text content, handwritten character content, table data, seal, date, and corresponding recognition box coordinate information, which provides standardized data for subsequent processing. In this embodiment, the recognition effect of the visual large model is also evaluated according to the accuracy rate of the recognized fields. The character accuracy rate evaluation formula is as follows:

[0076]

[0077] wherein, is the number of correctly recognized characters, is the total number of characters.

[0078] 3. Extract key information: input the text, recognition box, table, and other information recognized by the visual large model into a language large model, and write prompt words according to the business. The language large model extracts the corresponding key information according to the prompt words, and outputs the extraction results in the form of structured data. The language large model information extraction process is as shown in Figure 3 , and specifically includes:

[0079] (1) The title, text, handwritten characters, tables, seals, dates, and bounding box information recognized by the visual large model are input into the language large model.

[0080] (2) According to business needs, write targeted prompt words. Prompt words need to clearly specify the type of key information required, output format (such as JSON, CSV), and special requirements (such as data unit conversion, missing value processing method).

[0081] (3) The language large model reasons according to the prompt words and extracts the corresponding key information from the information recognized by the visual large model. Using attention mechanisms and pre-trained semantic understanding capabilities, it accurately locates and extracts data fields that meet the requirements.

[0082] (4) The extracted key information is structured and organized according to the preset format, and the standardized data is output. For example, the contract key information is organized into a JSON object containing "contract name", "signing date", "party A", "party B", "contract amount", etc. fields, which is convenient for subsequent storage, analysis and application.

[0083] 4. Check rule configuration: including building check scripts and check rule knowledge base. This embodiment designs a flexible and efficient checking method through the prompt word technology of the language large model and the extensible check rule knowledge base. This method supports the description of the association between the extracted content and the check rules through natural language prompt words, and the prompt words can be dynamically adjusted to adapt to different document types and business needs. And the general check rules and industry-specific specifications (such as industry standards) are stored in the check rule knowledge base, when the large model executes the check task, automatically retrieves the rule fragment with the highest matching degree in the knowledge base, and fuses the business context for reasoning. For example, in the contract checking scenario, the RAG retrieval enhancement technology extracts the latest clauses from the regulation library and compares them with the contract clauses for compliance; in the financial statement processing, historical data and accounting standards are called to verify the accuracy of data logic. The check rule configuration process is shown in Figure 4 , which specifically includes:

[0084] (1) Configure the check rules sorted according to business needs.

[0085] (2) Store the general check rules and industry-specific specifications (such as industry standards) in the check rule knowledge base R, and convert the check rule knowledge base into semantic vectors stored in the vector database , so as to facilitate the matching degree calculation of input data and rule vectors.

[0086]

[0087] wherein, represents each rule, Representation rules The number of words in the phrase.

[0088] (3) Configure a professional rule engine as the core carrier of rule execution and scheduling. Define rule execution priority to ensure that the review rule knowledge base (including industry standards, historical cases, etc.) is triggered first, followed by enterprise internal systems, to avoid rule conflicts that cause verification failure.

[0089] (4) Design prompt word templates through natural language, including four modules: task description (such as "check contract compliance"), knowledge reference, constraint conditions (such as a list of mandatory fields), and output format, to clearly define the review target and requirements.

[0090] (5) Based on business feedback (such as false positives and missed cases) or reinforcement learning algorithms, real-time adjustment of prompt word semantic expression and parameter settings can improve rule matching accuracy and enhance review accuracy through dynamic adjustment of prompt words.

[0091] 5. Review large model: structured data extracted by the language large model is input into the review large model, and combined with RAG retrieval enhancement generation technology, a dynamic double-engine retrieval mechanism is constructed: on the one hand, semantic similarity calculation is performed on the structured data to recall the highest matching basic review rules from the review rule knowledge base (including industry standards, historical cases, etc.); on the other hand, key word extraction and rule template matching can be used to trigger the rule engine to select business-specific verification logic from the review script. In this embodiment, if no corresponding review rule is found in the review rule knowledge base, the review rule is found through the review script. In the model reasoning stage, the review large model fuses business system data, recalled basic review rules, business-specific verification logic, and data to be reviewed, and performs in-depth verification from multiple dimensions such as logical consistency, format specification, and data accuracy. The review process of the review large model is shown in Figure 5 , and specifically includes:

[0092] (1) The key information extracted by the language large model is input into the review large model, and the review large model combines RAG retrieval enhancement generation technology to construct a dynamic double-engine retrieval mechanism.

[0093] (2) The semantic similarity of the key information extracted by the language large model and the vector database is calculated using the following formula.

[0094]

[0095] (3) According to the similarity score, the attention weight is calculated using the softmax function:

[0096] ​

[0097] (4) Then recall the highest matching basic review rules from the vector database (including industry standards, historical cases, etc.), the specific formula is as follows:

[0098]

[0099] wherein, is the retrieved document information, is the correlation between the first and the first retrieval result, is the set similarity threshold.

[0100] (5) Trigger the rule engine to filter business-specific verification logic from the review script through keyword extraction and rule template matching.

[0101] (6) In the model reasoning stage, the review large model fuses business system data, recalled basic review rules, business-specific verification logic, and data to be reviewed, and performs deep verification from multiple dimensions such as logical consistency, format specification, and data accuracy.

[0102] 6. Generate review results: By designing prompt words, guide the language large model to generate review results according to the reasoning results, the review results include but are not limited to: whether the document content meets the review rules prompt information; when there are situations that do not meet the review rules, specifically point out the error content, error type (such as data error, logical contradiction, format not standardized, etc.) and error position, and provide corresponding modification suggestions. The review results are displayed to the user in a visual form, through a pop-up window prompt, so that the user can quickly understand the problems in the document and make modifications.

[0103] This embodiment takes the intelligent payment agreement as an example to illustrate the method of the application.

[0104] 1. Obtain the document to be recognized, as shown in Figure 6 .

[0105] 2. Use the visual large model to analyze the document in multiple elements, which can accurately identify document titles, printed text paragraphs, handwritten content, tables, dates, seals, and other types of elements, and generate corresponding positioning frames for each identified element, as shown in Figure 7 .

[0106] 3. For table information, use the visual large model to identify the row and column structure of the table and the cell content, and convert it into editable table data. As shown in Table 1.

[0107] Table 1

[0108]

[0109] 4. For text information, handwritten content and date elements, accurately identify the order of the words and the hierarchy of the paragraphs. For seal elements, identify the text information in the seal. In combination with the identified table information, finally output all the information in the input picture as follows.

[0110] Intelligent payment agreement

[0111] Power supplier: Li's power (electricity) company

[0112] Power consumer: Power consumer name: Zhang's;

[0113] Power consumer user number: XXX;

[0114] Power consumer address: XX province, XX city, XX district, XX community.

[0115] In order to clarify the rights and obligations of the power consumer and the power supplier in the purchase and sale of electricity, according to the relevant provisions of "XXX", "XXX" and other relevant provisions, after consultation between the two parties, the intelligent payment agreement is reached as follows:

[0116] 1. The power consumer agrees to the power supplier's XXX management of the power consumer. XXXXXXXXXXXXXXXX and deduct the available electricity fee, and then calculate the available balance of the power consumer's electricity fee. When XXXXXXXXXXXX, the power supplier sends a reminder or XXX to the power consumer, reminding the power consumer to pay the fee in time; when XXXXXXXXXXXX, the power supplier will XXXX, and the power consumer will pay the fee after XXXX.

[0117] 2. The frequency of the power supplier's electricity fee calculation is XXX. The calculated copy of the electricity degree fee is XXXXXXXXXXXXXXXXXXXX.

[0118] 3. The power supplier calculates the electricity fee according to the determination of the power consumer. The power consumer can choose the second type of warning type according to the warning amount provided by the power supplier. (The power consumer can also choose whether to set a reminder amount, and the reminder amount is 20 yuan)

[0119]

[0120] 4. The power supplier sends the first opening XXXXXXXXXXXX notice to the power consumer in the form of XXX. The power consumer should specify the XXXX notice, and should keep the mobile phone open to receive the SMS notice. The mobile phone number for receiving the SMS notice is XXX.

[0121] 5. After the power is paid, when XXXX, choose the power restoration method: 1.

[0122] 1. The power user manually operates the closing button of the electric energy meter to restore power. 2. The electric energy meter automatically forces the closing to restore power.

[0123] Six, the power supply party measures the electricity fee every day, only XXXXXXXXX, and the actual electricity fee of the power user still XXXXXXXX. The power supply party XXXXXXX, according to the XXXX of the power user, calculates the actual electricity fee of the power user.

[0124] Seven, the responsibilities of both parties:

[0125] 1. The power supply party provides the power user with XXXX notice service, etc. If XXXX, the resulting loss is borne by the power supply party. If XXXX, etc. results in XXXX, the resulting loss is borne by the power user. If XXXXXXX, the power user will be XXXX after XXXX, the power supply party XXXX, XXXX and XXXX instructions.

[0126] 2. If the power user receives XXXX, he should change the procedures. If XXXX causes XXXX, all consequences are borne by XXXX.

[0127] 3. Due to XXXX or due to XXXX, resulting in XXXX.

[0128] 4. The power user should XXXX, if found abnormal, should XXXX E repair.

[0129] (The following is the text)

[0130] Power supply party (seal): XXXX Company Limited

[0131] Phone:

[0132] Signing time: June 23, 2025

[0133] Power user: Zhang

[0134] Legal representative (authorized agent):

[0135] Mobile phone: XXX

[0136] Signing time: June 23, 2025

[0137] 5. Input all information recognized by the visual big data model into the language big data model. Use the prompt word engineering to write the fields to be extracted, guiding the language big data model to extract keywords from the input information. For example, the fields to be extracted are: file name, power supplier name, power user name, power user ID, power user address, warning type, warning amount, power restoration method, seal information, contact number, and contract date. The warning amount is associated with the warning type. First, the visual big data model needs to know what the warning type is, and then it looks up the corresponding warning amount in the table information. Finally, output the required field information in a structured manner, as shown in Table 2.

[0138] Table 2

[0139]

[0140] 6. Input the extracted key information into the verification model, match it with some fields in the business system according to the pre-configured verification rules, and provide verification results and modification suggestions, such as... Figure 8 As shown.

[0141] Example 2

[0142] A document recognition and verification device based on a large model, comprising:

[0143] A data acquisition device is used to acquire the document to be identified and convert the document into an image format;

[0144] The first processing unit is used to input the obtained document images into a pre-trained large visual model and output the recognized information.

[0145] The second processing unit is used to input the information recognized by the visual big model into the language big model, write the field information to be extracted through prompt word engineering, guide the language big model to extract keywords from the input information, and output the field information to be extracted in a structured manner.

[0146] The third processing unit is used to configure verification rules, including building verification scripts and a verification rule knowledge base;

[0147] The fourth processing unit is used to input the data output by the second and third processing units into the large verification model. The large verification model combines RAG retrieval enhancement generation technology to construct a dynamic dual-engine retrieval mechanism to verify the input data.

[0148] The fifth processing unit is used to generate the verification results and display them visually.

[0149] Example 3

[0150] An electronic device includes a processor and a memory communicatively connected to the processor and configured to store instructions executable by the processor to perform the above-mentioned large model-based document identification and proofreading method.

[0151] Embodiment 4

[0152] A computer-readable storage medium storing a computer program, the computer program being executed by a processor to perform the above-mentioned large model-based document identification and proofreading method.

[0153] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the present disclosure comply with relevant legal regulations and do not violate public order and good customs.

[0154] It should be noted that personal information from users should be collected for legitimate and reasonable purposes, and should not be shared or sold outside these legitimate uses. In addition, such collection / sharing should be carried out after the user's informed consent is received, including but not limited to informing the user to read the user agreement / user notice before the user uses the function, and signing an agreement / authorization including authorization of relevant user information. In addition, any necessary steps should be taken to protect and secure access to such personal information data and ensure that other people with access to personal information data comply with their privacy and processes.

[0155] The present disclosure contemplates that user-selectable options for allowing or disabling access to personal information data can be provided to users. That is, the present disclosure contemplates that hardware and / or software components can be provided to prevent or detect unauthorized access to such personal information data. Where publically available personal information data are utilized for one or more implementations, the access to such data can be disabled. Further, such personal information data can be removed to protect the privacy of the users in accordance with the practices of the present disclosure. In addition, the personal information data can be de-identified to protect the privacy of the users.

[0156] The acquisition, transmission, storage, use, processing, etc. of data in the technical solutions of the present disclosure comply with relevant provisions of national laws and regulations.

[0157] It should be noted that in the embodiments of the present disclosure, some software, components, models, etc. of the prior art can be mentioned, which should be considered as exemplary, and the purpose is only to illustrate the feasibility of the technical solution implementation of the present application, but does not mean that the applicant has or will necessarily use the scheme.

[0158] In the foregoing descriptions of the embodiments, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0159] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0160] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain.

[0161] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered as a sequence of instructions to implement logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device, such as a computer-based system, processor- based system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a computer- readable storage medium or a computer-readable signal medium. The computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CD-ROM). In addition, the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

[0162] It should be understood that aspects of the present disclosure can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, the various steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. As such, if implemented in hardware and in another embodiment, any of the following technologies, or combinations thereof, can be used: discrete logic circuitry having logic gates for implementing logic functions upon data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and so forth.

[0163] Those of skill in the art would understand that the various steps or methods carried out by the above-mentioned embodiments can be carried out by program instructions executed by relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0164] In addition, each functional unit in each embodiment of the present disclosure can be integrated in one processing module, or each unit can exist physically separately, or two or more units can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software functional module. When the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0165] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present disclosure have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present disclosure.

Claims

1. A document recognition and verification method based on a large model, characterized in that: Includes the following steps: Step 1: Obtain the document to be identified and convert it into an image format; Step 2: Input the obtained document images into the pre-trained large visual model and output the recognized information; Step 3: Input the information recognized by the visual big model into the language big model, write the field information to be extracted through prompt word engineering, guide the language big model to extract keywords from the input information, and output the field information to be extracted in a structured manner; Step 4: Configure verification rules, including building verification scripts and a verification rule knowledge base; Step 5: Input the data output from Step 3 and Step 4 into the large verification model. The large verification model combines RAG retrieval enhancement generation technology to construct a dynamic dual-engine retrieval mechanism to verify the input data. Step 6: Generate the verification results and visualize them.

2. The document recognition and verification method based on a large model according to claim 1, characterized in that: Step 2 specifically includes: Collect document image datasets, perform image transformation on various types of document image data, and remove skewed and blurry document image data; The collected document images are labeled with polygonal annotations for text regions, table cells, and formula blocks. The field information to be extracted is also labeled. The annotation results are used to build a dataset, and training, validation, and test sets are generated through a custom script. The labeled document image data is fed into the large visual model, and the loss function is adjusted. The recognition results are fused in a multimodal manner and output as structured data, including the text content, text position, paragraph structure, table information, and corresponding bounding box coordinates in the image.

3. The document recognition and verification method based on a large model according to claim 1, characterized in that: Step 3 includes: The information recognized by the visual big model is input into the language big model; Write targeted prompts based on business needs; The language big model infers based on the prompt words, extracts corresponding key information from the information identified by the visual big model, and uses the attention mechanism and pre-trained semantic understanding ability to extract data fields that meet the requirements. The extracted key information is structured and organized according to a preset format to output standardized data.

4. The document recognition and verification method based on a large model according to claim 1, characterized in that: Step 4 includes: General verification rules and industry-specific standards are stored in the verification rule knowledge base; Configure a professional rule engine as the core carrier for rule execution and scheduling; The prompt template is designed using natural language and includes four main modules: task description, knowledge reference, constraints, and output format, which clearly define the verification objectives and requirements.

5. The document recognition and verification method based on a large model according to claim 4, characterized in that: Step 4 also includes: Based on business feedback or reinforcement learning algorithms, the semantic expression and parameter settings of prompt words are adjusted in real time.

6. The document recognition and verification method based on a large model according to claim 1, characterized in that: In step 5, the verification model, combined with RAG retrieval enhancement generation technology, constructs a dynamic dual-engine retrieval mechanism to verify the input information, including: Semantic similarity calculation is performed on the structured data, and the basic verification rule with the highest matching degree is recalled from the verification rule knowledge base; By extracting keywords and matching them with rule templates, the rules are triggered to select business-specific verification logic from the verification script; During the model inference phase, the verification big model integrates business system data, recall basic verification rules, business-specific verification logic, and data to be verified for inference, and performs in-depth verification from multiple dimensions.

7. A document recognition and verification device based on a large model, characterized in that: include: A data acquisition device is used to acquire the document to be identified and convert the document into an image format; The first processing unit is used to input the obtained document images into a pre-trained large visual model and output the recognized information. The second processing device is used to input the information recognized by the visual big model into the language big model, write the field information to be extracted through prompt word engineering, guide the language big model to extract keywords from the input information, and output the field information to be extracted in a structured manner. The third processing unit is used to configure verification rules, including building verification scripts and a verification rule knowledge base; The fourth processing unit is used to input the data output by the second and third processing units into the large verification model. The large verification model combines RAG retrieval enhancement generation technology to construct a dynamic dual-engine retrieval mechanism to verify the input data. The fifth processing unit is used to generate the verification results and display them visually.

8. An electronic device comprising a processor and a memory communicatively connected to the processor and used for storing processor-executable instructions, characterized in that: The processor is used to execute the method described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the method described in any one of claims 1-6.

Images