A large model-based document identification and checking method and device

Abstract

The application provides a large model-based document recognition and checking method and device. The method comprises the following steps: obtaining a document to be recognized; converting the document into a picture format; inputting the obtained document picture into a pre-trained visual large model; and outputting recognized information. The recognized information of the visual large model is input into a language large model. The field information to be extracted is written through a prompt word engineering, the language large model is guided to extract keywords from the input information, and the field information to be extracted is output in a structured manner. The checking rules are configured, including constructing a checking script and a checking rule knowledge base. The output data is input into a checking large model. The checking large model combines a RAG retrieval enhancement generation technology to construct a dynamic double-engine retrieval mechanism, checks the input information, generates a checking result, and performs visual display. The application can improve the accuracy and robustness of various document recognition.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing technology, and in particular relates to a document recognition and verification method and apparatus based on a large model. Background Technology

[0002] In the daily operations and project implementation of various industries, a large number of complex and highly specialized documents are generated and used. Traditional document processing methods mainly rely on manual identification and verification, requiring professionals to review each page. This approach is not only inefficient but also easily affected by subjective factors, making it difficult to meet the stringent requirements of various industries for efficiency and accuracy in document processing.

[0003] However, in the current field of document processing, traditional document recognition and verification methods mainly rely on Optical Character Recognition (OCR) technology and pre-defined rule-based verification algorithms. While OCR technology can extract text from images in documents, its accuracy is low when dealing with complex layouts, blurred handwriting, or handwritten text, and it is prone to missed or false recognitions. Furthermore, pre-defined rule-based verification algorithms require manually writing a large number of rules for different types of documents and verification needs, which is not only time-consuming and labor-intensive but also lacks flexibility and versatility, making it difficult to cope with diverse changes in document format and content.

[0004] With the rapid development of large model technology, its powerful language understanding and reasoning capabilities have opened up new possibilities for document processing. However, there is currently no mature solution that can effectively apply large model technology to the entire process of document recognition and verification, truly achieving efficient, accurate, and adaptive document processing. This also means that how to fully explore the potential of large model technology in the field of document processing remains a key issue that urgently needs to be addressed. Summary of the Invention

[0005] In view of this, the present invention aims to overcome the shortcomings of the above-mentioned problems in the prior art, and proposes a document recognition and verification method and device based on a large model. By utilizing the powerful capabilities of the large model, the accuracy and robustness of document recognition of various types can be improved, while realizing automated and intelligent document verification, reducing the cost of manual intervention, and improving the efficiency and quality of document processing.

[0006] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0007] In a first aspect, the present invention provides a document recognition and verification method based on a large model, comprising the following steps:

[0008] Step 1: Obtain the document to be recognized and convert it into an image format;

[0009] Step 2: input the obtained document image into a pre-trained visual large model, and output the recognized information;

[0010] Step 3: input the information recognized by the visual large model into a language large model, write the required field information through prompt word engineering, guide the language large model to extract keywords from the input information, and output the required field information in a structured manner;

[0011] Step 4: configure review rules, including building review scripts and review rule knowledge base;

[0012] Step 5: input the data output by steps 3 and 4 into a review large model, and the review large model combines RAG retrieval enhancement generation technology to build a dynamic double-engine retrieval mechanism to review the input data;

[0013] Step 6: generate review results and visualize them.

[0014] Further, step 2 specifically includes:

[0015] Collect document image data sets, and convert multiple types of document image data to images, remove skewed and blurred document image data;

[0016] Annotate the collected document images, and perform polygon annotation on text areas, table cells, and formula block elements. Also, annotate the field information to be extracted. Build the annotation results into a data set, and generate training set, validation set, and test set through custom scripts;

[0017] Pass the annotated document image data into the visual large model and adjust the loss function;

[0018] Fuse the recognition results in multiple modalities to output the recognition results in the form of structured data, including picture text content, text position, paragraph structure, table information, and corresponding recognition box coordinate information.

[0019] Further, step 3 includes:

[0020] Input the information recognized by the visual large model into the language large model;

[0021] Write targeted prompts according to business needs;

[0022] The language large model reasons according to the prompts, extracts the corresponding key information from the information recognized by the visual large model, and uses the attention mechanism and pre-trained semantic understanding ability to extract the required data field;

[0023] Structure the extracted key information according to the preset format and output standardized data.

[0024] Further, step 4 includes:

[0025] Store general check rules and industry-specific specifications in the check rule knowledge base;

[0026] Configure a professional rule engine as the core carrier of rule execution and scheduling;

[0027] Design prompt word templates through natural language, including four modules of task description, knowledge reference, constraint condition and output format, to clearly define the checking target and requirements.

[0028] Further, step 4 also includes:

[0029] Based on business feedback or reinforcement learning algorithm, real-time adjustment of prompt word semantic expression and parameter setting.

[0030] Further, in step 5, the check large model combines RAG retrieval enhancement generation technology to build a dynamic double-engine retrieval mechanism, and the checking of the input information includes:

[0031] Calculate the semantic similarity of structured data, and recall the highest matching basic check rule from the check rule knowledge base;

[0032] Through keyword extraction and rule template matching, trigger rules to filter business-specific verification logic from the dynamic rule library;

[0033] In the model inference stage, the check large model fuses business system data, recalled basic check rules, business-specific verification logic and data to be checked for inference, and performs deep checking from multiple dimensions.

[0034] In a second aspect, the present application provides a document recognition and checking device based on a large model, comprising:

[0035] A data acquisition device for acquiring a document to be identified and converting the document into a picture format;

[0036] A first processing device for inputting the obtained document picture into a pre-trained visual large model and outputting the recognized information;

[0037] A second processing device for inputting the information recognized by the visual large model into a language large model, writing the field information to be extracted through prompt word engineering, guiding the language large model to extract keywords from the input information, and outputting the field information to be extracted in a structured manner;

[0038] A third processing device for configuring check rules, including building check scripts and a check rule knowledge base;

[0039] The fourth processing device is configured to input the data output by the second processing device and the third processing device into a verification large model, and the verification large model is configured to perform verification on the input data by combining a RAG retrieval enhancement technology to construct a dynamic double-engine retrieval mechanism.

[0040] The fifth processing device is configured to generate a verification result and perform visual display.

[0041] In a third aspect, the present application provides an electronic device, which comprises a processor and a memory connected with the processor and configured to store executable instructions of the processor, and the processor is configured to execute the above-mentioned large model-based document identification and verification method.

[0042] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is configured to execute the above-mentioned large model-based document identification and verification method when executed by a processor.

[0043] Compared with the prior art, the large model-based document identification and verification method and device have the following advantages:

[0044] The present application innovatively combines visual large models (document image understanding) and language large models (semantic analysis) to realize end-to-end conversion from unstructured documents to structured data. It breaks through the limitation of traditional OCR that can only identify text content and simultaneously analyzes complex features such as layout structure, table logical relationship, and paragraph semantic association.

[0045] In the large model verification stage, the present application constructs a dynamic double-engine retrieval mechanism. On the one hand, it calculates the semantic similarity to recall the highest matching degree of the basic verification rules, and on the other hand, it extracts keywords and matches the business-specific verification logic through the rule template. This mechanism can flexibly combine general rules and business-specific rules to improve the accuracy and adaptability of verification. It can quickly adjust the verification logic according to different business scenarios, avoiding the limitations of a single rule base, reducing manual intervention, and improving verification efficiency.

[0046] In the verification process, the present application introduces RAG retrieval enhancement technology. For example, in the contract verification scenario, the latest provisions are extracted from the regulation library to combine with the contract provisions for compliance comparison, and in the financial statement processing, historical data is called to verify the accuracy of the data logic of the accounting standards. By obtaining external authoritative information in real time, the basis and reliability of the verification are enhanced, and the verification result is ensured to comply with the latest regulations and industry standards.

[0047] The present application stores general verification rules and industry-specific specifications (such as legal provisions and industry standards) in a knowledge base, realizes dynamic updating and expansion of verification rules, ensures that the verification process always complies with the latest industry standards and regulations, improves the guarantee of document compliance, and reduces legal risks.

[0048] The application closely combines the recognition process and the checking process of the document, realizes the integrated operation of recognition and checking. The user does not need to switch between multiple steps, and can complete the recognition and checking of the document in one process, which significantly improves the work efficiency and reduces the errors and delays in information transmission.

[0049] The application designs a prompt word strategy to guide the checking large model to generate a checking result. The generated checking result not only includes prompt information of whether the document content conforms to the checking rule, but also specifically points out the error content, error type, error position when there is a situation that does not conform to the checking rule, and provides corresponding modification suggestions. At the same time, the checking result is displayed to the user in a visual form, and the user is prompted through a pop-up window, so that the user can quickly understand the problems existing in the document and make modifications. BRIEF DESCRIPTION OF DRAWINGS

[0050] The accompanying drawings, which form a part of the present application, are included to provide a further understanding of the application, and are incorporated herein for purposes of illustrating the preferred embodiments of the present application and explaining the principles of the application. In the drawings:

[0051] Figure 1 The accompanying drawings, which form a part of the present application, are included to provide a further understanding of the application, and are incorporated herein for purposes of illustrating the preferred embodiments of the present application and explaining the principles of the application. In the drawings:

[0052] Figure 2 The accompanying drawings, which form a part of the present application, are included to provide a further understanding of the application, and are incorporated herein for purposes of illustrating the preferred embodiments of the present application and explaining the principles of the application. In the drawings:

[0053] Figure 3 The accompanying drawings, which form a part of the present application, are included to provide a further understanding of the application, and are incorporated herein for purposes of illustrating the preferred embodiments of the present application and explaining the principles of the application. In the drawings:

[0054] Figure 4 The accompanying drawings, which form a part of the present application, are included to provide a further understanding of the application, and are incorporated herein for purposes of illustrating the preferred embodiments of the present application and explaining the principles of the application. In the drawings:

[0055] Figure 5 The accompanying drawings, which form a part of the present application, are included to provide a further understanding of the application, and are incorporated herein for purposes of illustrating the preferred embodiments of the present application and explaining the principles of the application. In the drawings:

[0056] Figure 6 The accompanying drawings, which form a part of the present application, are included to provide a further understanding of the application, and are incorporated herein for purposes of illustrating the preferred embodiments of the present application and explaining the principles of the application. In the drawings:

[0057] Figure 7 The accompanying drawings, which form a part of the present application, are included to provide a further understanding of the application, and are incorporated herein for purposes of illustrating the preferred embodiments of the present application and explaining the principles of the application. In the drawings:

[0058] Figure 8 The accompanying drawings, which form a part of the present application, are included to provide a further understanding of the application, and are incorporated herein for purposes of illustrating the preferred embodiments of the present application and explaining the principles of the application. In the drawings: DETAILED DESCRIPTION

[0059] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0060] The present application will be described in detail below with reference to the accompanying drawings and embodiments.

[0061] Embodiment 1:

[0062] As Figure 1 shown, the present application provides a large model-based document recognition and checking method, the specific steps are as follows:

[0063] 1. Document picture acquisition: Obtain the picture of paper document through scanning equipment, photographing equipment, etc., or directly obtain the picture format file of electronic document as the original data to be processed.

[0064] 2. Visual large model identifies pictures: input the obtained document picture into the pre-trained visual large model, the visual large model is trained by a large amount of document image data and has the recognition ability of various document layouts, characters and symbols. The visual large model analyzes the document picture, identifies the text content, text position, paragraph structure, table information, etc. in the picture, supports multi-element recognition, and finally outputs the recognition result in the form of structured data. For example, for a document picture containing a table, the visual large model can identify the row and column structure of the table and the cell content, and convert them into editable table data; for paragraph text, it can accurately identify the text order and paragraph hierarchy, analyze the logical relationship between sentences in the paragraph, such as cause and effect relationship, parallel relationship, progressive relationship, etc. By identifying logical conjunctions (such as “therefore”, “but”, “first”, “second”, etc.) and contextual semantics, the logical structure of the paragraph is constructed. Finally, the recognized title, text, handwritten characters, table, seal, date and recognition box information, etc. are output. The visual large model identification process is shown in Figure 2 , which specifically includes:

[0065] (1) Collect real document image data set, convert multiple types of data into images, remove skewed and blurred data. Apply Laplacian operator to the image , calculate the variance of the image to evaluate the blur degree:

[0066]

[0067] wherein, is the total number of image pixels, is the average value of image pixels after Laplace transform. When is lower than a certain threshold , the image is determined to be blurred and removed.

[0068] (2) Label the collected document images, and perform polygon labeling on text regions, table cells, formula blocks, and other elements. Finally, construct the labeling results into a dataset, and generate training / validation / test sets (ratio 8:1:1) through a custom script. The purpose is to effectively evaluate the model performance when training the model, and prevent overfitting. The specific division formula is as follows:

[0069]

[0070] wherein, is the total data volume of the screened images.

[0071] (3) The labeled document image data is transmitted into a visual large model, and the loss function is fine-tuned. The specific formula is as follows:

[0072]

[0073] wherein, θ is a model parameter, f(x i ; θ) is a model output, is a true label. LoRA technology is used for lightweight fine-tuning to reduce training cost. A low-rank matrix is referenced to reduce training cost. The LoRA fine-tuning formula is as follows:

[0074]

[0075] (4) The recognition results are fused in multiple modalities, and output in the form of structured data. The output content includes recognized title content, text content, handwritten character content, table data, seal, date, and corresponding recognition box coordinate information, which provides standardized data for subsequent processing. In this embodiment, the recognition effect of the visual large model is also evaluated according to the accuracy rate of the recognized fields. The character accuracy rate evaluation formula is as follows:

[0076]

[0077] wherein, is the number of correctly recognized characters, is the total number of characters.

[0078] 3. Extract key information: input the text, recognition box, table, and other information recognized by the visual large model into a language large model, and write prompt words according to the business. The language large model extracts the corresponding key information according to the prompt words, and outputs the extraction results in the form of structured data. The language large model information extraction process is as shown in Figure 3 , and specifically includes:

[0079] (1) The title, text, handwritten characters, tables, seals, dates, and bounding box information recognized by the visual large model are input into the language large model.

[0080] (2) According to business needs, write targeted prompt words. Prompt words need to clearly specify the type of key information required, output format (such as JSON, CSV), and special requirements (such as data unit conversion, missing value processing method).

[0081] (3) The language large model reasons according to the prompt words and extracts the corresponding key information from the information recognized by the visual large model. Using attention mechanisms and pre-trained semantic understanding capabilities, it accurately locates and extracts data fields that meet the requirements.

[0082] (4) The extracted key information is structured and organized according to the preset format, and the standardized data is output. For example, the contract key information is organized into a JSON object containing "contract name", "signing date", "party A", "party B", "contract amount", etc. fields, which is convenient for subsequent storage, analysis and application.

[0083] 4. Check rule configuration: including building check scripts and check rule knowledge base. This embodiment designs a flexible and efficient checking method through the prompt word technology of the language large model and the extensible check rule knowledge base. This method supports the description of the association between the extracted content and the check rules through natural language prompt words, and the prompt words can be dynamically adjusted to adapt to different document types and business needs. And the general check rules and industry-specific specifications (such as industry standards) are stored in the check rule knowledge base, when the large model executes the check task, automatically retrieves the rule fragment with the highest matching degree in the knowledge base, and fuses the business context for reasoning. For example, in the contract checking scenario, the RAG retrieval enhancement technology extracts the latest clauses from the regulation library and compares them with the contract clauses for compliance; in the financial statement processing, historical data and accounting standards are called to verify the accuracy of data logic. The check rule configuration process is shown in Figure 4 , which specifically includes:

[0084] (1) Configure the check rules sorted according to business needs.

[0085] (2) Store the general check rules and industry-specific specifications (such as industry standards) in the check rule knowledge base R, and convert the check rule knowledge base into semantic vectors stored in the vector database , so as to facilitate the matching degree calculation of input data and rule vectors.

[0086]

[0087] wherein, represents each rule, Representation rules The number of words in the phrase.

[0088] (3) Configure a professional rule engine as the core carrier of rule execution and scheduling. Define rule execution priority to ensure that the review rule knowledge base (including industry standards, historical cases, etc.) is triggered first, followed by enterprise internal systems, to avoid rule conflicts that cause verification failure.

[0089] (4) Design prompt word templates through natural language, including four modules: task description (such as "check contract compliance"), knowledge reference, constraint conditions (such as a list of mandatory fields), and output format, to clearly define the review target and requirements.

[0090] (5) Based on business feedback (such as false positives and missed cases) or reinforcement learning algorithms, real-time adjustment of prompt word semantic expression and parameter settings can improve rule matching accuracy and enhance review accuracy through dynamic adjustment of prompt words.

[0091] 5. Review large model: structured data extracted by the language large model is input into the review large model, and combined with RAG retrieval enhancement generation technology, a dynamic double-engine retrieval mechanism is constructed: on the one hand, semantic similarity calculation is performed on the structured data to recall the highest matching basic review rules from the review rule knowledge base (including industry standards, historical cases, etc.); on the other hand, key word extraction and rule template matching can be used to trigger the rule engine to select business-specific verification logic from the review script. In this embodiment, if no corresponding review rule is found in the review rule knowledge base, the review rule is found through the review script. In the model reasoning stage, the review large model fuses business system data, recalled basic review rules, business-specific verification logic, and data to be reviewed, and performs in-depth verification from multiple dimensions such as logical consistency, format specification, and data accuracy. The review process of the review large model is shown in Figure 5 , and specifically includes:

[0092] (1) The key information extracted by the language large model is input into the review large model, and the review large model combines RAG retrieval enhancement generation technology to construct a dynamic double-engine retrieval mechanism.

[0093] (2) The semantic similarity of the key information extracted by the language large model and the vector database is calculated using the following formula.

[0094]

[0095] (3) According to the similarity score, the attention weight is calculated using the softmax function:

[0096] ​

[0097] (4) Then recall the highest matching basic review rules from the vector database (including industry standards, historical cases, etc.), the specific formula is as follows:

[0098]

[0099] wherein, is the retrieved document information, is the correlation between the first and the first retrieval result, is the set similarity threshold.

[0100] (5) Trigger the rule engine to filter business-specific verification logic from the review script through keyword extraction and rule template matching.

[0101] (6) In the model reasoning stage, the review large model fuses business system data, recalled basic review rules, business-specific verification logic, and data to be reviewed, and performs deep verification from multiple dimensions such as logical consistency, format specification, and data accuracy.

[0102] 6. Generate review results: By designing prompt words, guide the language large model to generate review results according to the reasoning results, the review results include but are not limited to: whether the document content meets the review rules prompt information; when there are situations that do not meet the review rules, specifically point out the error content, error type (such as data error, logical contradiction, format not standardized, etc.) and error position, and provide corresponding modification suggestions. The review results are displayed to the user in a visual form, through a pop-up window prompt, so that the user can quickly understand the problems in the document and make modifications.

[0103] This embodiment takes the intelligent payment agreement as an example to illustrate the method of the application.

[0104] 1. Obtain the document to be recognized, as shown in Figure 6 .

[0105] 2. Use the visual large model to analyze the document in multiple elements, which can accurately identify document titles, printed text paragraphs, handwritten content, tables, dates, seals, and other types of elements, and generate corresponding positioning frames for each identified element, as shown in Figure 7 .

[0106] 3. For table information, use the visual large model to identify the row and column structure of the table and the cell content, and convert it into editable table data. As shown in Table 1.

[0107] Table 1

[0108]

[0109] 4. For text information, handwritten content and date elements, accurately identify the order of the text and the hierarchy of the paragraphs. For seal elements, identify the text information in the seal. In combination with the identified table information, the final output of all information in the input picture is as follows.

[0110] Intelligent payment agreement

[0111] Power supplier: Li's power (electricity) company

[0112] Power consumer: Power consumer name: Zhang's;

[0113] Power consumer user number: XXX;

[0114] Power consumer address: XX province, XX city, XX district, XX community.

[0115] In order to clarify the rights and obligations of the power consumer and the power supplier in the purchase and sale of electricity, according to the relevant provisions of "XXX", "XXX" and other relevant regulations, after consultation between the two parties, the intelligent payment agreement is reached as follows:

[0116] 1. The power consumer agrees to the power supplier's XXX management of the power consumer. XXXXXXXXXXXXXXXX and deduct the available electricity fee, and then calculate the available balance of the power consumer. When XXXXXXXXXXXX, the power supplier sends a reminder or XXX to the power consumer, reminding the power consumer to pay the fee in time; when XXXXXXXXXXXX, the power supplier will XXXX, and the power consumer will pay the fee after XXXX.

[0117] 2. The frequency of the power supplier's electricity calculation is XXX. The calculated copy of the electricity degree is XXXXXXXXXXXXXXXXXXXX.

[0118] 3. The power supplier calculates the electricity fee according to the determination of the power consumer. The power consumer can choose the second type of warning type according to the warning amount provided by the power supplier. (The power consumer can also choose whether to set a warning amount, and the warning amount is 20 yuan)

[0119]

[0120] 4. The power supplier sends the first opening XXXX notice to the power consumer in the form of XXXX. The power consumer should specify the XXXX notice, and should keep the mobile phone open to receive the SMS notice. The mobile phone number for receiving the SMS notice is XXX.

[0121] 5. After paying the electricity fee, when XXXX, choose the power restoration method: 1.

[0122] 1. The power consumer manually operates the closing button of the electric energy meter to restore power. 2. The electric energy meter will automatically force the closing of the power supply.

[0123] Six, the power supply party calculates the electricity fee every day, only XXXXXXXX, and the electricity fee actually incurred by the electricity user every month is still XXXXXXXX. The power supply party XXXXXXX, according to the XXXX actually used by the electricity user, calculates the electricity fee actually incurred by the electricity user.

[0124] Seven, the responsibilities of both parties:

[0125] 1. The power supply party provides the electricity user with XXXX notice services, etc. If XXXX, the resulting loss will be borne by the power supply party. If XXXX, etc. results in XXXX, the resulting loss will be borne by the electricity user. If XXXXXXX, the electricity user will be instructed by the power supply party after XXXX.

[0126] 2. If the electricity user receives XXXX, the change procedures should be completed. If XXXX results in XXXX, all consequences will be borne by the electricity user.

[0127] 3. Due to XXXX or due to XXXX, resulting in XXXX.

[0128] 4. The electricity user should XXXX, and if any abnormalities are found, the electricity user should report for repair.

[0129] (The following is the main text)

[0130] Power supply party (seal): XXXX Company Limited

[0131] Phone:

[0132] Signing time: June 23, 2025

[0133] Electricity user: Zhang Xianxian

[0134] Legal representative (authorized agent):

[0135] Mobile phone number: XXX

[0136] Signing time: June 23, 2025

[0137] 5. Input all information recognized by the visual big data model into the language big data model. Use the prompt word engineering to write the fields to be extracted, guiding the language big data model to extract keywords from the input information. For example, the fields to be extracted are: file name, power supplier name, power user name, power user ID, power user address, warning type, warning amount, power restoration method, seal information, contact number, and contract date. The warning amount is associated with the warning type. First, the visual big data model needs to know what the warning type is, and then it looks up the corresponding warning amount in the table information. Finally, output the required extracted field information in a structured manner, as shown in Table 2.

[0138] Table 2

[0139]

[0140] 6. Input the extracted key information into the verification model, match it with some fields in the business system according to the pre-configured verification rules, and provide verification results and modification suggestions, such as... Figure 8 As shown.

[0141] Example 2

[0142] A document recognition and verification device based on a large model, comprising:

[0143] A data acquisition device is used to acquire the document to be identified and convert the document into an image format;

[0144] The first processing unit is used to input the obtained document images into a pre-trained large visual model and output the recognized information.

[0145] The second processing unit is used to input the information recognized by the visual big model into the language big model, write the field information to be extracted through prompt word engineering, guide the language big model to extract keywords from the input information, and output the field information to be extracted in a structured manner.

[0146] The third processing unit is used to configure verification rules, including building verification scripts and a verification rule knowledge base;

[0147] The fourth processing unit is used to input the data output by the second and third processing units into the large verification model. The large verification model combines RAG retrieval enhancement generation technology to construct a dynamic dual-engine retrieval mechanism to verify the input data.

[0148] The fifth processing unit is used to generate the verification results and display them visually.

[0149] Example 3

[0150] An electronic device comprises a processor and a memory connected to the processor in communication, and the memory is configured to store instructions executable by the processor, and the processor is configured to execute the above-mentioned large model-based document identification and checking method.

[0151] Embodiment 4

[0152] A computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the above-mentioned large model-based document identification and checking method.

[0153] The collection, storage, use, processing, transmission, provision and disclosure of user personal information in the present disclosure comply with relevant laws and regulations and do not violate public order and good customs.

[0154] It should be noted that the personal information from the user should be collected for legal and reasonable purposes, and should not be shared or sold outside these legal uses. In addition, such collection / sharing should be carried out after the user's informed consent is received, including but not limited to informing the user to read the user agreement / user notice before the user uses the function, and signing the agreement / authorization including authorization of relevant user information. In addition, any necessary steps should be taken to protect and secure access to such personal information data and ensure that other people with access to personal information data comply with their privacy and procedures.

[0155] The present disclosure contemplates providing embodiments in which the user can selectively block the use or access of personal information data. That is, the present disclosure contemplates providing hardware and / or software to prevent or block access to such personal information data. Minimizing the risk of data collection and deletion of data can minimize risks once the personal information data is no longer needed. In addition, such personal information is de-identified to protect the privacy of the user, when applicable.

[0156] The acquisition, transmission, storage, use, processing, etc. of data in the technical solutions of the present disclosure comply with the relevant provisions of national laws and regulations.

[0157] It should be noted that in the embodiments of the present disclosure, some existing industry solutions may be mentioned, such as certain software, components, models, etc. They should be considered as exemplary, and their purpose is only to illustrate the feasibility of the technical solutions in the present application. However, it does not mean that the applicant has or will necessarily use the solution.

[0158] In the foregoing descriptions of the embodiments, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0159] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0160] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain.

[0161] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered as a sequence of instructions to implement logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device, such as a computer-based system, processor- based system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a computer- readable storage medium or a computer-readable signal medium. The computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CD-ROM). In addition, the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

[0162] It should be understood that aspects of the present disclosure can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, the various steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. As such, if implemented in hardware and in another embodiment, any of the following technologies, known in the art, or their combinations can be used to implement the hardware: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0163] Those skilled in the art can understand that all or part of the steps carried out by the above-mentioned embodiment method can be completed by programs instructing related hardware, and the programs can be stored in a computer-readable storage medium. When the programs are executed, one or a combination of the steps of the method embodiment is included.

[0164] In addition, each functional unit in each embodiment of the present disclosure can be integrated in one processing module, or each unit can exist physically separately, or two or more units can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software functional module. When the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0165] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present disclosure have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present disclosure.

Claims

1. A large model-based document identification and proofreading method, characterized in that: Comprise the following steps: Step 1: Obtain the document to be identified, and convert the document into a picture format; Step 2: input the obtained document picture into the pre-trained visual large model, and output the recognized information; specifically comprising: Collecting document image data sets, image converting multiple types of document image data, removing skewed and blurred document image data; Labeling the collected document images, polygon labeling text areas, table cells, and formula block elements, and labeling the field information to be extracted, constructing the labeling results into a data set, and generating a training set, a validation set, and a test set through a self-defined script; The labeled document image data is transmitted into the visual large model, and the loss function is adjusted; Multi-modal fusion of the recognition result is output in the form of structured data, including the picture text content, text position, paragraph structure, table information, and corresponding recognition box coordinate information; Step 3: input the information recognized by the visual large model into the language large model, write the field information to be extracted through prompt word engineering, guide the language large model to extract keywords from the input information, and output the field information to be extracted in a structured manner; Step 4: configure the review rules, including building review scripts and review rule knowledge base; store general review rules and industry-specific specifications in the review rule knowledge base; specifically comprising: Configure the professional rule engine as the core carrier of rule execution and scheduling; Design the prompt word template through natural language, including four modules of task description, knowledge reference, constraint condition, and output format, to clearly define the review target and requirements; Step 5: input the data output by the step 3 and the step 4 into the review large model, and the review large model combines the RAG retrieval enhancement generation technology to build a dynamic double-engine retrieval mechanism to review the input data; specifically comprising: The review large model combines the RAG retrieval enhancement generation technology to build a dynamic double-engine retrieval mechanism to review the input information, including: Calculate the semantic similarity of structured data, and recall the basic review rules with the highest matching degree from the review rule knowledge base; Through keyword extraction and rule template matching, trigger rules to filter business-specific verification logic from the review script; In the model inference stage, the review large model fuses business system data, recalled basic review rules, business-specific verification logic, and data to be reviewed for inference, and performs deep verification from multiple dimensions; Step 6: generate review results and perform visual display.

2. The method of claim 1, wherein the method further comprises: The step 3 comprises: Input the information recognized by the visual large model into the language large model; According to the business needs, write targeted prompt words; The language large model reasons according to the prompt words, extracts the corresponding key information from the information recognized by the visual large model, uses the attention mechanism and the pre-trained semantic understanding ability to extract the required data field; The extracted key information is structured and arranged according to the preset format, and the standardized data is output.

3. The method of claim 1, wherein the method further comprises: The step 4 further comprises: Based on business feedback or reinforcement learning algorithm, the semantic expression of prompt words and parameter settings are adjusted in real time.

4. A large model-based document recognition and proofreading device, characterized by: The method comprises the following steps: The data acquisition device is used to acquire a document to be recognized and convert the document into a picture format. The first processing device is used to input the obtained document picture into a pre-trained visual large model and output recognized information; specifically comprising: Collecting document image data sets, converting multiple types of document image data, removing skewed and blurred document image data; Labeling the collected document images, performing polygon labeling on text regions, table cells, and formula block elements, and labeling the field information to be extracted, constructing the labeling results into a data set, and generating a training set, a validation set, and a test set through a self-defined script; The labeled document image data is transmitted into the visual large model, and the loss function is adjusted; The recognition result is fused in multiple modalities to output the recognition result in the form of structured data, including the text content, text position, paragraph structure, table information, and corresponding recognition box coordinate information in the picture; The second processing device is used to input the information recognized by the visual large model into a language large model, write the field information to be extracted through prompt word engineering, guide the language large model to extract keywords from the input information, and output the field information to be extracted in a structured manner; The third processing device is used to configure review rules, including constructing review scripts and a review rule knowledge base; specifically comprising: Configuring a professional rule engine as the core carrier of rule execution and scheduling; Designing a prompt word template through natural language, including four modules of task description, knowledge reference, constraint condition, and output format, to clearly define the review target and requirements; The fourth processing device is used to input the data output by the second processing device and the third processing device into a review large model, and the review large model combines RAG retrieval enhancement generation technology to construct a dynamic double-engine retrieval mechanism to review the input data; specifically comprising: The review large model combines RAG retrieval enhancement generation technology to construct a dynamic double-engine retrieval mechanism to review the input information, comprising: Calculating the semantic similarity of structured data, recalling the basic review rules with the highest matching degree from the review rule knowledge base; Triggering the rule to select business-specific verification logic from the review script through keyword extraction and rule template matching; In the model reasoning stage, the review large model fuses business system data, recalled basic review rules, business-specific verification logic, and data to be reviewed for reasoning, and performs deep verification from multiple dimensions; The fifth processing device is used to generate review results and perform visual display.

5. An electronic device, comprising a processor and a memory connected to the processor in communication, and configured to store executable instructions of the processor, characterized in that: The processor is used to execute the method of any one of claims 1-3.

6. A computer readable storage medium storing a computer program, characterized in that: The computer program is executed by the processor to implement the method of any one of claims 1-3.

Images