Document auditing method and device and electronic equipment

Abstract

The invention discloses a document auditing method and device and electronic equipment. The method comprises the following steps: acquiring a to-be-audited document; feature extraction is carried out on the to-be-audited document and a preset auditing rule to obtain document features and rule features, the preset auditing rule is an auditing standard when the to-be-audited document is audited, and the document features at least comprise text features, visual features and layout features corresponding to the to-be-audited document; the rule features are used for representing key information features in the preset auditing rule; according to the document features and the rule features, a target modal corresponding to the to-be-audited document is determined, and the target modal is a modal type adopted when the to-be-audited document is audited; and analyzing the to-be-audited document according to the target modality and a preset auditing rule to obtain a document auditing result. The technical problems that due to the fact that modal selection is single and fusion is insufficient, document understanding is not comprehensive, and auditing accuracy and efficiency are limited are solved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and more specifically, to a document review method, device, and electronic device. Background Art

[0002] With the rapid development of information technology, document review, as a key link in information processing and management, has attracted widespread attention for its efficiency and accuracy. Traditional document review methods, especially for documents such as project applications and policy implementation in the government sector, rely primarily on manual review. This not only consumes a large amount of human resources, but also has low review efficiency and is easily influenced by personal experience and subjective judgment, resulting in inconsistent review standards and varying accuracy. The limitations of manual review are particularly evident when dealing with documents containing large numbers of tables, images, and complex layouts. It is often difficult to fully capture and understand all the information in the document, especially those non-textual visual and layout features that are critical to ensuring the compliance and consistency of document content.

[0003] To address this challenge, automated document review technology has made significant progress in recent years. For example, RAG (Retrieval-Augmented Generation) can search for valid information related to review tasks in large-scale text databases by integrating retrieval and generation technologies, thereby generating more targeted review results. However, RAG technology in related technologies mainly focuses on the processing of text information, ignoring the visual and layout features in the document. These features are indispensable for the review of government documents because they not only provide additional contextual information, but also often carry proof of the correctness and compliance of key data.

[0004] In addition, in the relevant technology, although there is a document audit solution that converts the document into an image format and then uses OCR (Optical Character Recognition) technology to extract text information, this method is still limited to the analysis of text content and fails to fully consider the impact of layout structure and visual elements on the audit results. When encountering documents with inconsistent formats, complex layouts, or containing non-standard fonts and images, simple text analysis often leads to the omission or misunderstanding of important information, which in turn affects the accuracy and completeness of the audit. In addition, since different types of business documents (such as engineering construction project applications, policy subsidy applications, etc.) have their own specific audit requirements, a single modal processing method is difficult to meet the audit requirements in diverse scenarios, reducing the applicability and effectiveness of the automatic audit system.

[0005] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0006] The embodiments of the present application provide a document review method, device, and electronic device to at least solve the technical problems of incomplete document understanding, limited review accuracy, and efficiency caused by single modality selection and insufficient fusion.

[0007] According to one aspect of an embodiment of the present application, a document review method is provided, comprising: obtaining a document to be reviewed; performing feature extraction on the document to be reviewed and preset review rules respectively to obtain document features and rule features, wherein the preset review rules are review standards when reviewing the document to be reviewed, and the document features include at least text features, visual features and layout features corresponding to the document to be reviewed, and the rule features are used to represent key information features in the preset review rules; determining a target modality corresponding to the document to be reviewed based on the document features and the rule features, wherein the target modality is the modality type adopted when reviewing the document to be reviewed; analyzing the document to be reviewed based on the target modality and the preset review rules to obtain a document review result.

[0008] Optionally, before extracting features from the document to be reviewed, the method also includes: correcting the direction of the document to be reviewed if the direction deviation angle of the document to be reviewed is greater than a preset deviation value; removing the blank area in the document to be reviewed if there is a blank area in the document to be reviewed; and reducing the resolution of the document to be reviewed if the number of pages of the document to be reviewed exceeds a preset number of pages.

[0009] Optionally, the layout features include at least one of the following: a metadata summary, a content summary, a chart summary, and a layout summary of the document to be reviewed.

[0010] Optionally, after obtaining the document features and rule features, the method further includes: vectorizing the document features to obtain a document feature vector, wherein the document feature vector includes a first text feature vector corresponding to the text features in the document to be reviewed, a first visual feature vector corresponding to the visual features in the document to be reviewed, and a layout feature vector corresponding to the layout features in the document to be reviewed; vectorizing the rule features to obtain a rule feature vector, wherein the rule feature vector includes a second text feature vector corresponding to the text features in the preset review rules, and a second visual feature vector corresponding to the visual features in the preset review rules.

[0011] Optionally, determining a target modality corresponding to the document to be reviewed based on document features and rule features includes: determining a similarity sequence based on a document feature vector and a rule feature vector, wherein the similarity sequence includes a text similarity sequence between the document to be reviewed and the preset review rule in a text modality, a visual similarity sequence in a visual modality, and a layout similarity sequence in a layout modality; determining a modality weight corresponding to the similarity sequence, and determining a similarity score of the similarity sequence based on the modality weight, wherein the modality weight is used to represent the importance of modalities corresponding to different similarity sequences under the preset review rules; and determining the modality corresponding to the similarity sequence with the highest similarity score as the target modality of the document to be reviewed.

[0012] Optionally, determining a similarity sequence based on the document feature vector and the rule feature vector includes: determining a text similarity sequence based on the first text feature vector and the second text feature vector; determining a visual similarity sequence based on the first visual feature vector and the second visual feature vector; and determining a layout similarity sequence based on the layout feature vector.

[0013] Optionally, determining the modal weight corresponding to the similarity sequence includes: obtaining the metadata vector, content vector, chart vector and layout vector in the layout feature vector; determining the weight set corresponding to the similarity sequence based on the rule feature vector, metadata vector, content vector, chart vector and layout vector; and determining the weight corresponding to the maximum value in the weight set as the modal weight.

[0014] Optionally, the method further includes: determining the number of pages of the document to be reviewed; and when the number of pages is less than a preset threshold, determining a document review result of the document to be reviewed based on preset review rules and document features.

[0015] According to another aspect of the embodiment of the present application, a document review device is also provided, including: an acquisition module for acquiring a document to be reviewed; an extraction module for performing feature extraction on the document to be reviewed and preset review rules respectively, to obtain document features and rule features, wherein the preset review rules are the review standards when reviewing the document to be reviewed, and the document features include at least text features, visual features and layout features corresponding to the document to be reviewed, and the rule features are used to represent key information features in the preset review rules; a determination module for determining a target modality corresponding to the document to be reviewed based on the document features and rule features, wherein the target modality is the modality type adopted when reviewing the document to be reviewed; and a review module for analyzing the document to be reviewed based on the target modality and the preset review rules to obtain a document review result.

[0016] According to another aspect of the embodiments of the present application, an electronic device is provided, including: a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the above-mentioned document review method.

[0017] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned document review method by running the computer program.

[0018] According to another aspect of the embodiments of the present application, a computer program product is provided, including computer instructions, which implement the above-mentioned document review method when executed by a processor.

[0019] In an embodiment of the present application, a document to be reviewed is obtained; features of the document to be reviewed and preset review rules are extracted respectively to obtain document features and rule features, wherein the preset review rules are the review standards when reviewing the document to be reviewed, and the document features include at least text features, visual features and layout features corresponding to the document to be reviewed, and the rule features are used to represent the key information features in the preset review rules; the target modality corresponding to the document to be reviewed is determined based on the document features and rule features, wherein the target modality is the modality type adopted when reviewing the document to be reviewed; the document to be reviewed is analyzed based on the target modality and the preset review rules to obtain the document review result, thereby achieving the purpose of intelligent identification and optimal modality matching, thereby realizing the technical effect of automation, precision and efficiency of the document review process, and further solving the technical problems of incomplete document understanding, limited review accuracy and efficiency caused by single modality selection and insufficient fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0021] Figure 1 is a hardware structure diagram of a computer terminal for implementing a document review method according to an embodiment of the present application;

[0022] Figure 2 is a flowchart of a document review method according to an embodiment of the present application;

[0023] Figure 3 This is a structural diagram of a document review device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] First, some nouns or terms that appear in the process of explaining the embodiments of this application are subject to the following explanations:

[0027] Large Language Model (LLM): A deep learning model that, trained on massive amounts of text data, can understand and generate human language. This type of model can perform a variety of complex natural language processing tasks, including but not limited to text generation, translation, question answering, and sentiment analysis. Its core characteristic is its large parameter size, which enables it to capture the complex structure and context of language, thereby improving the accuracy and coherence of tasks.

[0028] VLMs (Visual Large Models) are deep learning models trained on image or video data that can recognize and understand visual content. These models can be used for tasks such as object recognition, scene parsing, image classification, and visual question answering. Similar to large language models, visual large models rely on large datasets for training to accurately parse and process complex visual information.

[0029] RAG (Retrieval-Augmented Generation): A hybrid model framework that combines retrieval and generation capabilities. RAG first uses a retrieval component to find the most relevant pieces of information for an input query in a large-scale text database. It then leverages a generation component to produce more accurate and specific answers or content based on the retrieved information. Designed to improve the performance of natural language generation tasks, RAG can generate more coherent and meaningful output, particularly when background knowledge or contextual information is required.

[0030] Optical Character Recognition (OCR): The process of converting text in images or scanned documents into machine-encoded text. OCR automatically identifies text content, enabling machines to read and process information in paper documents or images, playing a vital role in digital conversion and document management.

[0031] Layout analysis refers to the process of parsing and understanding a document's physical layout (e.g., font size, paragraphs, tables, image placement, etc.). For complex documents, such as government documents, layout analysis helps better locate and extract information, ensuring its completeness and accuracy. Layout analysis is a key step in identifying and parsing unstructured documents, particularly in scenarios like information extraction and data form processing.

[0032] In order to solve the problem of limited document review efficiency in the related art, the present application embodiment provides a document review method that can be run on Figure 1 Among the computer terminals shown, the computer terminal will be described below.

[0033] The document review method embodiment provided in the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal for implementing a document review method is shown in FIG. Figure 1 As shown, the computer terminal 10 may include one or more (illustrated by 102a, 102b, ..., 102n in the figure) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions connected via a wired and / or wireless network. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. It will be understood by those skilled in the art that Figure 1The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0034] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry." The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10. As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0035] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the document review method in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the above-mentioned document review method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0036] The transmission module 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission module 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission module 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0037] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0038] It should be noted that, in some optional embodiments, the above Figure 1The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0039] In the above-mentioned operating environment, an embodiment of the present application provides an embodiment of a document review method. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0040] Figure 2 is a flowchart of a document review method according to an embodiment of the present application. Figure 2 As shown, the method includes the following steps:

[0041] Step S202: Obtain the document to be reviewed.

[0042] In the above step S202, the document to be reviewed may come from various channels such as online submission, attachment to an email, or scanned upload of a physical document.

[0043] In the embodiments of the present application, the focus is mainly on document review in the government affairs field, such as project applications, policy implementation and other documents in this field, aiming to improve the accuracy and consistency of key information filling. Its goals are significantly different from traditional text review. Specifically, the traditional review mechanism mainly focuses on identifying and correcting typos and grammatical errors in documents, and ensuring that the content complies with established standards and policy requirements. However, the review mechanism in this application goes deep into the substantive content of the document, paying special attention to the accuracy of specific data in the project application documents, such as whether the time points of "intended start time" and "intended completion time" are reasonable and consistent with the project plan; whether the descriptions of "land use nature" and "land area" are consistent with the actual land use situation, and whether there are any misreports or omissions; in addition, it also focuses on the correctness of the project type and construction nature, ensuring that labels such as "investment approval project" and "new construction" in the project application form are accurately checked to avoid subsequent processing problems caused by information classification errors.

[0044] In step S204, feature extraction is performed on the document to be reviewed and the preset review rules respectively to obtain document features and rule features, wherein the preset review rules are the review standards when reviewing the document to be reviewed, and the document features include at least text features, visual features and layout features corresponding to the document to be reviewed, and the rule features are used to represent the key information features in the preset review rules.

[0045] In step S204, document features encompass three dimensions: textual, visual, and layout features, while rule features accurately describe the key elements of the audit standards. This step is implemented using deep learning and artificial intelligence technologies, such as using a text vector representation model to extract text features, a visual vector representation model to extract visual features, and layout analysis technology to identify layout features. The extraction of rule features involves natural language processing and semantic understanding of the audit standards, ensuring that key information points in the audit process, such as specific clauses, keywords, and standard formats, are captured.

[0046] In the embodiment of the present application, since different review rules will affect the recall strategy, all review rules can be classified using the few-shot method in the large language model to obtain the following categories of review rules:

[0047] 1. Information Existence: Pay attention to the completeness of key fields in the document to ensure that all necessary information has been provided. For example, whether the project applicant has signed the signature field, or whether necessary data such as the planned start date, planned completion date, land use nature, and land area have been filled in.

[0048] 2. Information is filled in correctly: This function verifies the logic and accuracy of the information in the document. For example, if the approval result delivery method is mail delivery, the system will verify whether the complete mailing address is provided and whether the project type and construction nature are checked to meet the preset conditions.

[0049] 3. Consistency between external information and completed information: This is used to compare the consistency of information within the document with external data. For example, check whether the application date is within the commission period and whether the project code is strictly consistent with external records to ensure that the document content is consistent with external facts.

[0050] 4. Layout related: In addition to the text content, the overall visual elements are reviewed, such as whether the seal is stamped as required to ensure the formality and legality of the document.

[0051] 5. Others: Special circumstances outside the above rules, leaving room for handling unforeseen audit needs.

[0052] Step S206 : determining a target modality corresponding to the document to be reviewed based on the document characteristics and the rule characteristics, wherein the target modality is the modality type adopted when reviewing the document to be reviewed.

[0053] In the above step S206, after the feature extraction is completed, the modal weight can be calculated based on the document features (mainly layout features) and rule features, and then the target modality can be determined. Among them, the calculation of the modal weight is a key operation. By analyzing the correlation between different modal features and audit rules, the importance of text, visual and layout modalities can be dynamically adjusted to adapt to different types of documents and audit requirements. The target modality ensures that during the audit process, the system can focus on the most relevant and influential modal types, thereby improving the accuracy and efficiency of the audit.

[0054] Step S208: Analyze the document to be reviewed based on the target modality and preset review rules to obtain a document review result.

[0055] In step S208, a large language model, a large visual model, or a layout analysis algorithm can be used to parse the document content based on the characteristics of the target modality, identify key information, verify compliance with the rules, and generate a final document review result. The document review result is presented in the form of structured data or a report, clearly indicating whether the document meets the review standards, as well as any discrepancies or issues, providing clear guidance for subsequent decision-making and revisions.

[0056] Through steps S202 to S208 above, the goal of intelligent identification and optimal modality matching is achieved, thereby achieving the technical effects of automating, accurately, and efficiently implementing the document review process. This further addresses the technical issues of incomplete document understanding, limited review accuracy, and efficiency caused by single modality selection and insufficient integration. This is explained in detail below.

[0057] In the above step S202, after obtaining the document to be reviewed, it also includes: when the direction deviation angle of the document to be reviewed is greater than the preset deviation value, correcting the direction of the document to be reviewed; when there are blank areas in the document to be reviewed, removing the blank areas in the document to be reviewed; when the number of pages of the document to be reviewed exceeds the preset number of pages, reducing the resolution of the document to be reviewed.

[0058] In an embodiment of the present application, after the document to be reviewed is obtained, it will first be converted into an image and the image will be preprocessed to ensure that the visual information and text information of the document can be optimally parsed and utilized by the model.

[0059] Specifically, because documents for review may be scanned or photographed in various ways, their orientation may deviate. If the system detects that a document's orientation deviates by an angle exceeding a preset value, it automatically performs orientation correction to ensure the document's reading direction is consistent with conventional readings, thereby preventing the visual model from misinterpreting the document's content.

[0060] Subsequently, the document is checked for blank areas, and these blank areas are automatically removed, retaining the valid information in the document, thereby improving the recognition accuracy and speed of key areas by the visual large model and ensuring efficient review.

[0061] Furthermore, the system will also appropriately reduce the document resolution, especially for long documents exceeding a preset number of pages, to account for the possibility of documents containing multiple pages. This moderately reduced resolution speeds up the large visual model's understanding while ensuring accurate recognition of text and image information, thereby improving overall document review efficiency.

[0062] In the above step S204, feature extraction is performed on the document to be reviewed to obtain document features, which may include the following steps:

[0063] S1: Text feature extraction.

[0064] OCR technology can be used to identify and extract text information from document images to form text features.

[0065] S2: Visual feature extraction.

[0066] Because document page content is complex and diverse, relying solely on layout analysis cannot fully perceive elements such as page color and overall layout. Therefore, global visual encoding technology can be used to capture the overall visual characteristics of the page, including color distribution and layout structure, to provide a more comprehensive understanding of the page.

[0067] S3: Layout feature extraction.

[0068] Optionally, the layout features include at least one of the following: a metadata summary, a content summary, a chart summary, and a layout summary of the document to be reviewed.

[0069] In this embodiment, layout feature extraction is designed to further refine the understanding of document structure and content. This process includes the generation of metadata summaries, content text, figure and table summaries (including sub-figure summaries and table summaries), and layout summary text, covering all components of the document and its visual representation.

[0070] Specifically, the metadata summary covers basic information such as the color and page number of the document. This information can be used as filtering conditions in the retrieval stage to improve the accuracy of the search. The global content summary ensures an overall grasp of the text content of the document and can be recognized using OCR technology. Sub-image summaries and table summaries can extract key information from charts and tables through a deep understanding of the visual large model to make up for the shortcomings of OCR technology in processing graphics and tables. It should be noted that since there may be embedded images in the table, a separate image summary will be generated on the basis of generating the table summary text and spliced to the original table summary. The layout summary focuses on non-standardized layout elements, such as handwriting and seals, which can provide additional certification basis for the review process through the recognition and understanding of the visual model.

[0071] Optionally, after obtaining the document features and rule features, it also includes: vectorizing the document features to obtain a document feature vector, wherein the document feature vector includes a first text feature vector corresponding to the text features in the document to be reviewed, a first visual feature vector corresponding to the visual features in the document to be reviewed, and a layout feature vector corresponding to the layout features in the document to be reviewed; vectorizing the rule features to obtain a rule feature vector, wherein the rule feature vector includes a second text feature vector corresponding to the text features in the preset review rules, and a second visual feature vector corresponding to the visual features in the preset review rules.

[0072] In the embodiments of this application, vectorization converts the original document features and rule features into numerical representations, enabling efficient calculation and comparison in mathematical space. A specific analysis can be as follows:

[0073] 1. Document feature vectorization.

[0074] Document feature vectorization is the process of converting the text features, visual features, and layout features of a document into corresponding feature vectors.

[0075] Specifically, the text content of the document will be converted into a first text feature vector. For example, a text embedding model (such as BERT, Conan, etc.) is used to capture the semantics and contextual information of the text and convert it into a high-dimensional vector to facilitate subsequent text matching and semantic understanding. Similarly, the visual information in the document, such as images, charts, etc., is converted into a first visual feature vector. For example, a large visual model (such as Colpali, ColQwen, VisRAG-Ret, LayoutLMv3, etc.) is used to parse visual elements such as objects, colors, and layouts in the image to generate vector representations that reflect these features. Due to the complexity and diversity of visual information, the vector representation must be rich enough to cover all visual details that may affect the audit results. Finally, the layout features of the document, including metadata summary, content summary, sub-image summary, table summary, and layout summary, are converted into layout feature vectors.

[0076] 2. Regular feature vectorization.

[0077] Rule feature vectorization involves converting preset audit rules, including textual and visual rules, into corresponding feature vectors.

[0078] Specifically, the text portion of the preset audit rules is converted into a second text feature vector, which can be processed using the same model as the document's text features, ensuring semantic consistency between the rules and the document's features. The visual portion of the preset audit rules is converted into a second visual feature vector, which can also be processed using the same model as the document's visual features.

[0079] In the above-mentioned step S206, the target modality corresponding to the document to be reviewed is determined based on the document features and the rule features, including: determining a similarity sequence based on the document feature vector and the rule feature vector, wherein the similarity sequence includes a text similarity sequence between the document to be reviewed and the preset review rule in the text modality, a visual similarity sequence in the visual modality, and a layout similarity sequence in the layout modality; determining a modality weight corresponding to the similarity sequence, and determining a similarity score of the similarity sequence based on the modality weight, wherein the modality weight is used to represent the importance of the modalities corresponding to different similarity sequences under the preset review rules; and determining the modality corresponding to the similarity sequence with the highest similarity score as the target modality of the document to be reviewed.

[0080] In the embodiment of this application, the modality selection strategy is the core of this method. By calculating the similarity between document features and rule features and determining the target modality based on these similarity scores, the accuracy and efficiency of document review are optimized. The specific steps can be as follows:

[0081] S1: Similarity sequence calculation.

[0082] Optionally, determining a similarity sequence based on the document feature vector and the rule feature vector includes: determining a text similarity sequence based on the first text feature vector and the second text feature vector; determining a visual similarity sequence based on the first visual feature vector and the second visual feature vector; and determining a layout similarity sequence based on the layout feature vector.

[0083] In an embodiment of the present application, a preliminary screening and recall is performed by using a preset audit rule (query), that is, the document feature vector and the rule feature vector are compared to obtain a similarity sequence under different modalities. Specifically, by calculating the similarity between the first text feature vector and the second text feature vector, a text similarity sequence Top1 under the text modality is formed: [sim11, sim12, sim1N]; by calculating the similarity between the first visual feature vector and the second visual feature vector, a visual similarity sequence Top2 under the visual modality is formed: [sim21, sim22, sim2N]; by calculating the matching degree between the layout feature vector and the layout requirements in the preset audit rule, a layout similarity sequence Top3 under the layout modality is formed: [sim31, sim32, sim3N].

[0084] These similarity sequences quantify the degree of match between a document and the audit rules in each modality, providing foundational data for the subsequent determination of modal weights and target modalities. For example, a text similarity sequence might reflect the results of an item-by-item comparison of the document's text content with the requirements of the rules; a visual similarity sequence might assess document compliance based on visual features like color and layout; and a layout similarity sequence focuses on the degree to which document structure and non-standard elements, such as seals and signatures, match the layout requirements of the rules.

[0085] S2: Modal weight determination.

[0086] Optionally, determining the modal weight corresponding to the similarity sequence includes: obtaining the metadata vector, content vector, chart vector and layout vector in the layout feature vector; determining the weight set corresponding to the similarity sequence based on the rule feature vector, metadata vector, content vector, chart vector and layout vector; and determining the weight corresponding to the maximum value in the weight set as the modal weight.

[0087] Specifically, the modal weight is determined by analyzing the metadata vector, content vector, graphic vector and layout vector in the layout feature vector and combining it with the rule feature vector. For each similarity sequence, a weight set is generated. This reflects the relative importance of different modalities under the current review rules. For example, when reviewing a document containing numerous charts and tables, the weights for chart vectors and layout vectors may be relatively high, as these visual and layout features are crucial for review decisions. By selecting the maximum value from the set of weights and assigning this highest weight as the modal weight for the current modality and review rules, the review process prioritizes the most relevant features, thereby improving review accuracy.

[0088] The specific expression is as follows:

[0089]

[0090] In the formula, m represents different modes, a m represents the modal weight under mode m, exp represents the exponential function, sim() represents the similarity function, Q t represents the regular feature vector, S t Represents the tth element vector in the layout feature vector, S k Represents the kth element vector in the layout feature vector, and k represents the number of element vectors in the layout feature vector.

[0091] Furthermore, the similarity sequence scores under different modalities can be determined based on the modal weights.

[0092] The specific expression is as follows:

[0093]

[0094] Where Score represents the similarity sequence score, that is, the review score of different modalities (text modality, visual modality and layout modality); Q m represents the regular eigenvector under mode m, D m Represents the document feature vector under modality m.

[0095] S3: Target modality selection.

[0096] Based on the similarity scores of the above similarity sequences and the corresponding modal weights, the modality corresponding to the similarity sequence with the highest similarity score is selected as the target modality. It is important to note that this selection is dynamic and varies depending on the specific document content and preset review rules. For example, for a document with rich visual features but less textual content, the system may select the visual modality as the target modality for more efficient review.

[0097] This adaptive modal selection mechanism enables the audit system to flexibly respond to various complex documents, whether they are text-intensive, chart-intensive, or have complicated layout designs. It can find the most appropriate audit perspective and improve the intelligence level of the audit process.

[0098] In the above step S208, it also includes: determining the page number of the document to be reviewed; when the page number is less than a preset threshold, determining the document review result of the document to be reviewed according to preset review rules and document features.

[0099] In the embodiments of this application, a flexible audit strategy is proposed, which uses different audit mechanisms based on the number of pages in the document to be audited. Specifically, when it is determined that the number of pages in the document to be audited is less than a preset threshold, a deep comparison is performed directly based on the preset audit rules, combining the document's textual, visual, and layout features to quickly generate an audit result. This direct audit method simplifies the processing flow, avoids additional modal fusion calculations, and significantly improves the speed and efficiency of auditing small or simple documents.

[0100] On the contrary, for large documents with page numbers that reach or exceed the preset threshold, a more sophisticated and comprehensive multimodal adaptive fusion review strategy will be adopted. Figure 2 The adaptive modality preference perception strategy in the document review process performs high-level content understanding and audit rule matching, ultimately generating document review results. This approach not only handles the diverse information and layout structures in complex documents, but also dynamically adjusts the weights of different modalities based on the characteristics of the audit rules, ensuring comprehensive and accurate audits.

[0101] In the embodiments of this application, by dynamically adjusting the weights of text, visual, and layout modalities, as well as a flexible and adaptable review process, documents of varying complexity can be reviewed efficiently and accurately. At the same time, by cleverly combining the advantages of large visual models and large language models, and utilizing a modality preference adaptive strategy, while ensuring comprehensive review, the use of computing resources is optimized, enabling intelligent capture and analysis of multi-dimensional document information. This allows for adaptive selection of the optimal review path at varying document sizes, greatly facilitating the work of reviewers and improving the accuracy and speed of document review.

[0102] According to an embodiment of the present application, a document review device is provided. It should be noted that the document review device of the embodiment of the present application can be used to execute the document review method provided in the embodiment of the present application. The document review device provided in the embodiment of the present application is introduced below.

[0103] Figure 3 This is a structural diagram of a document review device provided according to an embodiment of the present application. Figure 3 As shown, the device includes:

[0104] An acquisition module 30 is used to acquire documents to be reviewed;

[0105] Extraction module 32 is used to extract features from the document to be reviewed and the preset review rules, respectively, to obtain document features and rule features, wherein the preset review rules are the review standards for reviewing the document to be reviewed, and the document features include at least text features, visual features, and layout features corresponding to the document to be reviewed, and the rule features are used to represent key information features in the preset review rules;

[0106] A determination module 34 is configured to determine a target modality corresponding to the document to be reviewed based on the document characteristics and the rule characteristics, wherein the target modality is the modality type used when reviewing the document to be reviewed;

[0107] The review module 36 is used to analyze the document to be reviewed based on the target modality and preset review rules to obtain the document review result.

[0108] Through the acquisition module, extraction module, determination module and review module in the above-mentioned document review device, the purpose of intelligent identification and optimal modality matching is achieved, thereby realizing the technical effects of automation, precision and efficiency of the (government) document review process, and further solving the technical problems of incomplete document understanding, limited review accuracy and efficiency caused by single modality selection and insufficient integration.

[0109] In the document review device provided in the embodiment of the present application, the extraction module is also used to correct the direction of the document to be reviewed when the direction deviation angle of the document to be reviewed is greater than a preset deviation value; to remove the blank area in the document to be reviewed when there is a blank area in the document to be reviewed; and to reduce the resolution of the document to be reviewed when the number of pages of the document to be reviewed exceeds a preset number of pages.

[0110] In the document review device provided in the embodiment of the present application, the extraction module is also used to vectorize the document features to obtain a document feature vector, wherein the document feature vector includes a first text feature vector corresponding to the text features in the document to be reviewed, a first visual feature vector corresponding to the visual features in the document to be reviewed, and a layout feature vector corresponding to the layout features in the document to be reviewed; and vectorize the rule features to obtain a rule feature vector, wherein the rule feature vector includes a second text feature vector corresponding to the text features in the preset review rules, and a second visual feature vector corresponding to the visual features in the preset review rules.

[0111] In the document review device provided in the embodiment of the present application, the determination module is also used to determine a similarity sequence based on the document feature vector and the rule feature vector, wherein the similarity sequence includes a text similarity sequence between the document to be reviewed and the preset review rule in the text modality, a visual similarity sequence in the visual modality, and a layout similarity sequence in the layout modality; determine the modal weight corresponding to the similarity sequence, and determine the similarity score of the similarity sequence based on the modal weight, wherein the modal weight is used to indicate the importance of the modalities corresponding to different similarity sequences under the preset review rules; and determine the modality corresponding to the similarity sequence with the highest similarity score as the target modality of the document to be reviewed.

[0112] In the document review device provided in the embodiment of the present application, the determination module is also used to determine the text similarity sequence based on the first text feature vector and the second text feature vector; determine the visual similarity sequence based on the first visual feature vector and the second visual feature vector; and determine the layout similarity sequence based on the layout feature vector.

[0113] In the document review device provided in the embodiment of the present application, the determination module is also used to obtain the metadata vector, content vector, chart vector and layout vector in the layout feature vector; determine the weight set corresponding to the similarity sequence based on the rule feature vector, metadata vector, content vector, chart vector and layout vector; and determine the weight corresponding to the maximum value in the weight set as the modal weight.

[0114] In the document review device provided in the embodiment of the present application, the review module is also used to determine the number of pages of the document to be reviewed; when the number of pages is less than a preset threshold, the document review result of the document to be reviewed is determined based on the preset review rules and document characteristics.

[0115] An embodiment of the present application also provides an electronic device, including: a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the above-mentioned document review method.

[0116] It should be noted that the above electronic equipment is used to perform Figure 2 The document review method shown in the figure, therefore the relevant explanations in the above document review method also apply to this electronic device and will not be repeated here.

[0117] An embodiment of the present application also provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned document review method by running the computer program.

[0118] It should be noted that the above non-volatile storage medium is used to execute Figure 2The document review method shown in the figure, therefore the relevant explanations in the above document review method are also applicable to the non-volatile storage medium and will not be repeated here.

[0119] An embodiment of the present application also provides a computer program product, including computer instructions, which implement the above-mentioned document review method when executed by a processor.

[0120] It should be noted that the above-mentioned computer program product is used to execute Figure 2 The document review method shown, therefore the relevant explanations in the above document review method also apply to this computer program product and will not be repeated here.

[0121] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0122] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0123] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0124] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0125] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0126] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0127] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A document review method, characterized in that: include: Get documents for review; Performing feature extraction on the document to be reviewed and the preset review rules respectively to obtain document features and rule features, wherein the preset review rules are review standards for reviewing the document to be reviewed, the document features at least include text features, visual features, and layout features corresponding to the document to be reviewed, and the rule features are used to represent key information features in the preset review rules; Determining a target modality corresponding to the document to be reviewed based on the document characteristics and the rule characteristics, wherein the target modality is a modality type adopted when reviewing the document to be reviewed; The document to be reviewed is analyzed according to the target modality and the preset review rules to obtain a document review result.

2. The method according to claim 1, characterized in that Before extracting features from the document to be reviewed, the method further includes: When the direction deviation angle of the document to be reviewed is greater than a preset deviation value, correcting the direction of the document to be reviewed; If there is a blank area in the document to be reviewed, removing the blank area in the document to be reviewed; When the number of pages of the document to be reviewed exceeds a preset number of pages, the resolution of the document to be reviewed is reduced.

3. The method according to claim 1, characterized in that The layout features include at least one of the following: a metadata summary, a content summary, a chart summary, and a layout summary of the document to be reviewed.

4. The method according to claim 1, wherein After obtaining the document features and the rule features, the method further includes: Performing vectorization processing on the document features to obtain a document feature vector, wherein the document feature vector includes a first text feature vector corresponding to the text features in the document to be reviewed, a first visual feature vector corresponding to the visual features in the document to be reviewed, and a layout feature vector corresponding to the layout features in the document to be reviewed; The rule features are vectorized to obtain a rule feature vector, wherein the rule feature vector includes a second text feature vector corresponding to the text feature in the preset review rule and a second visual feature vector corresponding to the visual feature in the preset review rule.

5. The method according to claim 4, characterized in that Determining a target modality corresponding to the document to be reviewed based on the document characteristics and the rule characteristics includes: Determining a similarity sequence based on the document feature vector and the rule feature vector, wherein the similarity sequence includes a text similarity sequence between the document to be reviewed and the preset review rule in a text mode, a visual similarity sequence in a visual mode, and a layout similarity sequence in a layout mode; Determining modal weights corresponding to the similarity sequences, and determining similarity scores for the similarity sequences based on the modal weights, wherein the modal weights are used to indicate the importance of modalities corresponding to different similarity sequences under the preset review rules; The modality corresponding to the similarity sequence with the highest similarity score is determined as the target modality of the document to be reviewed.

6. The method according to claim 5, characterized in that Determining a similarity sequence based on the document feature vector and the rule feature vector includes: determining the text similarity sequence based on the first text feature vector and the second text feature vector; determining the visual similarity sequence according to the first visual feature vector and the second visual feature vector; The layout similarity sequence is determined based on the layout feature vector.

7. The method according to claim 5, characterized in that Determining a modality weight corresponding to the similarity sequence includes: Obtaining metadata vectors, content vectors, chart vectors, and layout vectors from the layout feature vector; Determining a weight set corresponding to the similarity sequence based on the rule feature vector, the metadata vector, the content vector, the chart vector, and the layout vector; The weight corresponding to the maximum value in the weight set is determined as the modal weight.

8. The method according to claim 1, characterized in that The method further comprises: Determining the number of pages of the document to be reviewed; When the number of pages is less than a preset threshold, the document review result of the document to be reviewed is determined according to the preset review rule and the document feature.

9. A document review device, characterized in that: include: Acquisition module, used to obtain documents to be reviewed; an extraction module for extracting features from the document to be reviewed and the preset review rules, respectively, to obtain document features and rule features, wherein the preset review rules are review standards for reviewing the document to be reviewed, the document features at least including text features, visual features, and layout features corresponding to the document to be reviewed, and the rule features are used to represent key information features in the preset review rules; a determination module, configured to determine a target modality corresponding to the document to be reviewed based on the document characteristics and the rule characteristics, wherein the target modality is a modality type adopted when reviewing the document to be reviewed; The review module is used to analyze the document to be reviewed according to the target modality and the preset review rules to obtain a document review result.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store program instructions; The processor is connected to the memory and is used to execute the document review method described in any one of claims 1 to 8.

11. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the document review method according to any one of claims 1 to 8 by running the computer program.

12. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the document review method according to any one of claims 1 to 8 is implemented.

Images