Document processing method and device based on multi-modal feature fusion and electronic equipment

Abstract

Embodiments of the present application provide a document processing method and device based on multi-modal feature fusion and electronic equipment, relating to the technical field of intelligent processing of financial documents. The method comprises: after obtaining a financial document image, jointly extracting text information, image features and semantic information, evaluating the reliability of the text based on the multi-modal analysis result, correcting low-confidence content, fusing the image features with the corrected semantic analysis result to generate structured data, and then combining the financial knowledge graph to perform semantic contradiction detection, risk classification and field correction, and finally outputting the result. The method improves the information extraction accuracy, logical verification capability and risk identification effect, and enhances the stability and compliance of financial document processing.

Description

Technical Field

[0001] This application relates to the field of intelligent processing technology for financial documents, and in particular to a document processing method, apparatus and electronic device based on multimodal feature fusion. Background Technology

[0002] In financial business scenarios such as account opening, credit granting, and contract processing, existing technologies typically use optical character recognition (OCR) to extract text information from documents, and then human reviewers compare and make logical judgments on each extracted text item by item.

[0003] However, existing technologies mostly rely on character extraction and manual comparison, making it difficult to take into account the layout structure information in document images, the semantic context of text, and the relationships between multiple documents. This leads to misidentification of fields, loss of structural information, and missed detection of logical contradictions, affecting the efficiency and accuracy of the review process.

[0004] Therefore, how to improve the accuracy of information extraction and logical verification in the process of financial document processing, while enhancing the ability to identify potential risks and adapt to rule changes, has become an urgent technical problem to be solved. Summary of the Invention

[0005] This application provides a document processing method, apparatus, and electronic device based on multimodal feature fusion to solve the aforementioned technical problems. This method is designed for financial business document processing scenarios. It constructs a unified processing mechanism from information extraction and result verification to anomaly correction, focusing on the collaborative utilization of document image information, text content information, and business semantic relationships. While improving the accuracy of document field recognition, it enhances the comprehensive judgment capability of structural information, semantic relationships, and business logic consistency, thereby improving the accuracy, stability, and risk identification level of financial document processing.

[0006] In a first aspect, embodiments of this application provide a document processing method based on multimodal feature fusion, comprising:

[0007] Acquire a document image of the financial document to be processed, and perform character recognition on the document image to obtain text data;

[0008] Image feature analysis is performed on the document image, and semantic parsing is performed on the text data;

[0009] Based on the image feature analysis results and semantic parsing results, the confidence score of the text data is determined;

[0010] When the confidence score is lower than a preset threshold, the text data is corrected using a natural language processing model.

[0011] The image feature analysis results are fused with the analysis results obtained by re-semantically analyzing the corrected text data to generate structured data;

[0012] Semantic contradiction detection is performed on the structured data based on financial knowledge graphs to determine contradictory fields and their contradiction risk levels.

[0013] Based on the corresponding conflict risk level, the conflict field is corrected to obtain the corrected structured data, which is then output.

[0014] In one possible implementation, determining the confidence score of the text data based on the image feature analysis results and semantic parsing results includes:

[0015] Based on the layout structure information in the semantic parsing results and the layout structure information in the image feature analysis results, the layout matching degree is calculated;

[0016] The confidence score of the text data is determined based on the character confidence score output during character recognition and the layout matching degree.

[0017] In one possible implementation, calculating the layout matching degree based on the layout structure information in the semantic parsing result and the layout structure information in the image feature analysis result includes:

[0018] Based on the semantic parsing results, determine the fields corresponding to each target element in the text data;

[0019] Extract the image coordinate regions of each target element in the document image from the image feature analysis results;

[0020] Based on a preset financial document template, determine the preset coordinate region corresponding to each target element;

[0021] For each target element, calculate the distance deviation between the image coordinate region of the target element and the preset coordinate region;

[0022] The distance deviation value is converted into a layout matching score according to a preset mapping rule;

[0023] The layout matching degree is calculated based on the layout matching degree sub-score of each target element.

[0024] In one possible implementation, determining the confidence score of the text data based on the character confidence score output during character recognition and the layout matching degree includes:

[0025] The text data is segmented according to the target element to obtain at least one target element to be scored;

[0026] For each target element, obtain the character confidence score, the corresponding layout matching score, and the preset character confidence score weight and layout matching score weight.

[0027] Based on the character confidence weight and the layout matching weight, the character confidence and layout matching sub-scores are weighted and summed to obtain the comprehensive confidence sub-score of the target element.

[0028] The confidence score of the text data is obtained by weighted averaging of the comprehensive confidence scores of each target element.

[0029] In one possible implementation, the step of correcting the text data using a natural language processing model includes:

[0030] Extract target elements from the text data whose confidence scores are lower than a preset threshold;

[0031] Locate the semantic anchor point of the target element in the document, wherein the semantic anchor point includes at least one of the field name of the target element, adjacent target elements, or table header;

[0032] Based on the contextual information of semantic anchors and combined with financial business logic rules, the correct value of the target element is inferred.

[0033] The incorrect value in the target element is replaced with the inferred correct value to obtain the corrected text data.

[0034] In one possible implementation, the financial knowledge graph includes financial business entities, entity attributes, relationships between entities, and constraint rules. The step of performing semantic contradiction detection on the structured data based on the financial knowledge graph to determine contradictory fields and their contradiction risk levels includes:

[0035] For each field in the structured data, the target entity and entity attributes corresponding to each field are retrieved from the financial knowledge graph;

[0036] Based on the relationships and constraint rules in the financial knowledge graph, a consistency check is performed on the target entity and its attributes.

[0037] If the verification result is inconsistent, the corresponding contradictory field and contradiction type in the structured data are located based on the inconsistent entity attributes, and the risk level of the contradiction is determined according to the preset risk level mapping rules.

[0038] In one possible implementation, the step of correcting the contradictory field according to the corresponding contradictory risk level to obtain and output the corrected structured data includes:

[0039] If the risk level is low risk, then the preset rule base is invoked, the corresponding correction rule is matched according to the contradiction type, the correction value is obtained, and the error value in the contradiction field is replaced with the correction value to obtain the corrected structured data;

[0040] If the risk level is high risk, an early warning report is generated containing the original value of the contradiction field, the suggested correction value, the contradiction type description, and the correction suggestion, and the early warning report is pushed to the review terminal.

[0041] In one possible implementation, the method further includes:

[0042] When the risk level is high risk, receive the confirmed correction report returned by the audit terminal;

[0043] Based on the correction values ​​in the correction report, the contradictory fields are corrected to obtain corrected structured data, which is then output.

[0044] Secondly, embodiments of this application provide a document processing apparatus based on multimodal feature fusion, comprising:

[0045] The acquisition module is used to acquire document images of the financial documents to be processed;

[0046] The processing module is used to perform character recognition on the document image to obtain text data;

[0047] The processing module is also used to perform image feature analysis on the document image and semantic parsing on the text data;

[0048] The processing module is also used to determine the confidence score of the text data based on the image feature analysis results and semantic parsing results;

[0049] The processing module is also used to correct the text data using a natural language processing model when the confidence score is lower than a preset threshold.

[0050] The processing module is further configured to fuse the image feature analysis results with the parsing results obtained by re-semantically parsing the corrected text data to generate structured data.

[0051] The processing module is also used to perform semantic contradiction detection on the structured data based on the financial knowledge graph, and to determine the contradictory fields and their contradiction risk levels.

[0052] The processing module is also used to correct the contradiction field according to the corresponding contradiction risk level, and output the corrected structured data.

[0053] In one possible implementation, the device further includes: a calculation module and a determination module;

[0054] The calculation module is used to calculate the layout matching degree based on the layout structure information in the semantic parsing result and the layout structure information in the image feature analysis result;

[0055] The determination module is used to determine the confidence score of the text data based on the character confidence score output during character recognition and the layout matching degree.

[0056] In one possible implementation, the determining module is further configured to determine the fields corresponding to each target element in the text data based on the semantic parsing result;

[0057] The processing module is also used to extract the image coordinate regions of each target element in the document image from the image feature analysis results;

[0058] The determining module is further configured to determine the preset coordinate region corresponding to each target element based on a preset financial document template;

[0059] The calculation module is also used to calculate the distance deviation between the image coordinate region of the target element and the preset coordinate region for each target element;

[0060] The processing module is also used to convert the distance deviation value into a layout matching sub-score according to a preset mapping rule;

[0061] The calculation module is also used to calculate the layout matching degree based on the layout matching degree sub-score of each target element.

[0062] In one possible implementation, the processing module is further configured to segment the text data according to the target element to obtain at least one target element to be scored;

[0063] The acquisition module is also used to acquire, for each target element, the character confidence score, the corresponding layout matching score, and the preset character confidence score weight and layout matching score weight.

[0064] The calculation module is further configured to perform a weighted summation calculation on the character confidence score and the layout matching score sub-score based on the character confidence score weight and the layout matching score weight, so as to obtain the comprehensive confidence score sub-score of the target element.

[0065] The processing module is also used to perform a weighted average of the comprehensive confidence sub-scores of each target element to obtain the confidence score of the text data.

[0066] In one possible implementation, the processing module is further configured to extract target elements in the text data whose confidence scores are lower than a preset threshold;

[0067] The processing module is also used to locate the semantic anchor point of the target element in the document, wherein the semantic anchor point includes at least one of the field name of the target element, adjacent target elements, or table header;

[0068] The processing module is also used to infer the correct value of the target element based on the context information of the semantic anchor and in combination with financial business logic rules.

[0069] The processing module is also used to replace the incorrect value in the target element with the inferred correct value to obtain the corrected text data.

[0070] In one possible implementation, the processing module is further configured to search for the target entity and entity attributes corresponding to each field in the financial knowledge graph for the fields in the structured data;

[0071] The processing module is also used to perform consistency verification on the target entity and entity attributes based on the associations and constraint rules in the financial knowledge graph.

[0072] The determining module is further configured to, if the verification result is inconsistent, locate the corresponding contradictory field and contradiction type in the structured data based on the inconsistent entity attributes, and determine the risk level of the contradiction according to the preset risk level mapping rules.

[0073] In one possible implementation, the processing module is further configured to, if the risk level is low risk level, call a preset rule base, match the corresponding correction rule according to the contradiction type, obtain a correction value, and replace the error value in the contradiction field with the correction value to obtain the corrected structured data;

[0074] The processing module is further configured to generate an early warning report containing the original value of the contradiction field, a suggested correction value, a description of the contradiction type, and a correction suggestion if the risk level is high risk level, and push the early warning report to the review terminal.

[0075] In one possible implementation, the processing module is further configured to receive a confirmed correction report returned by the audit terminal when the risk level is high risk level.

[0076] The processing module is further configured to correct the contradictory field according to the correction value in the correction report, and output the corrected structured data.

[0077] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0078] The memory stores computer-executed instructions;

[0079] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0080] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0081] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0082] The document processing method, apparatus, and electronic device based on multimodal feature fusion provided in this application acquire document images of financial documents to be processed and perform character recognition to obtain text data. Image feature analysis is performed on the document images, and semantic parsing is performed on the text data. A confidence score for the text data is determined based on the image feature analysis results and the semantic parsing results. When the confidence score is lower than a preset threshold, the text data is corrected using a natural language processing model. The image feature analysis results are fused with the parsing results obtained from re-semantically parsing the corrected text data to generate structured data. Semantic contradiction detection is performed on the structured data based on a financial knowledge graph, contradictory fields and their contradiction risk levels are determined, and contradictory fields are corrected according to the corresponding contradiction risk levels. This comprehensively utilizes image features, text semantics, and domain knowledge to verify and correct the field content, structural relationships, and logical consistency in financial documents, thereby improving the accuracy of information extraction and logical verification in financial documents, enhancing the ability to identify anomalies, contradictions, and potential risks, and improving the efficiency of document review and processing. Attached Figure Description

[0083] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0084] Figure 1 A flowchart illustrating the document processing method based on multimodal feature fusion provided in this application. Figure 1 ;

[0085] Figure 2 A flowchart illustrating the document processing method based on multimodal feature fusion provided in this application. Figure 2 ;

[0086] Figure 3 A flowchart illustrating the document processing method based on multimodal feature fusion provided in this application. Figure 3 ;

[0087] Figure 4 A schematic diagram of the document processing device based on multimodal feature fusion provided in this application;

[0088] Figure 5 A schematic diagram of the structure of the electronic device provided in this application.

[0089] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0090] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.

[0091] First, let me explain the terms used in this application:

[0092] OCR (Optical Character Recognition) refers to the technical process of automatically recognizing printed or handwritten text in an image and converting it into text data that can be edited and processed by a computer using computer vision technology.

[0093] Intelligent financial document processing technology is commonly used in banks, consumer finance companies, insurance institutions, and other business scenarios requiring identity verification, credit review, and contract archiving. Customers typically submit various types of document images, such as identity documents, income statements, loan application materials, bank statements, and signed contracts, through methods like counter terminals, self-service devices, mobile phone photo uploads, or batch imports from the business backend. The backend then identifies, reviews, and stores these documents. Unlike general text processing, financial documents not only contain textual information such as amounts, dates, identification numbers, and signatories, but also structural information at the image level, such as seal locations, signature areas, table layouts, and relative field positions. Therefore, the processing typically involves the collaboration of multiple stages, including image acquisition, character recognition, business review, and document storage.

[0094] In existing technologies, document images are typically uploaded to a central processing platform by front-end acquisition devices. The platform then extracts the text before handing it over to manual review and anomaly detection. High-frequency business requires both the ability to extract content from a single document and the ability to support consistency comparisons between multiple documents. Therefore, it faces the characteristics of large processing volume and high accuracy requirements. Once an identification error or logical verification is missing, it may directly affect the approval result.

[0095] In view of this, how to improve the accuracy of information extraction and logical verification in financial document processing, while taking into account risk identification capabilities and compliance adaptability, has become an urgent technical problem to be solved.

[0096] To address the aforementioned issues, this application provides a document processing method based on multimodal feature fusion. After acquiring the document image of the financial document to be processed, this method first performs character recognition on the document image to obtain text data, and simultaneously performs image feature analysis on the document image and semantic parsing on the text data. Then, based on the image feature analysis results and semantic parsing results, a confidence score for the text data is determined. When the confidence score is lower than a preset threshold, the text data is corrected using a natural language processing model. Based on this, the image feature analysis results are fused with the parsing results obtained from re-semantically parsing the corrected text data to generate structured data. Furthermore, semantic contradiction detection is performed on the structured data based on a financial knowledge graph to determine contradictory fields and their contradiction risk levels. The contradictory fields are then corrected according to the corresponding contradiction risk levels to obtain corrected structured data, which is then output. By combining image features, text semantics, confidence score, knowledge graph verification, and risk level correction, the impact of field misidentification and missing structural information is reduced, improving the ability to detect logical contradictions and the targeted nature of corrections, making the financial document processing process more accurate.

[0097] This method can be deployed on edge computing nodes or cloud servers within a financial institution's business system. Edge computing nodes include dedicated business servers located within the bank's intranet, intelligent terminal devices with image processing capabilities, or local processing gateways that interface with the banking business system. It obtains text data by acquiring document images of financial documents and performing character recognition. Image feature analysis is then performed on the document images, and semantic parsing is conducted on the text data. A confidence score for the text data is determined based on the image feature analysis and semantic parsing results. When the confidence score is below a preset threshold, the text data is corrected using a natural language processing model. The image feature analysis results and the semantic parsing results of the corrected text data are fused to generate structured data. Semantic contradiction detection is performed on the structured data based on a financial knowledge graph, and contradictory fields and their risk levels are identified. Contradictory fields are then graded and corrected according to their risk levels, and the corrected structured data is output. This achieves intelligent closed-loop processing of financial documents throughout the entire process, from information extraction, logical verification, risk identification to automatic correction.

[0098] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0099] Figure 1 A flowchart illustrating the document processing method based on multimodal feature fusion provided in this application. Figure 1 ,like Figure 1 As shown, the method includes:

[0100] S101. Obtain the document image of the financial document to be processed, and perform character recognition on the document image to obtain text data.

[0101] Financial documents refer to various paper or electronic documents used in financial transactions (such as account opening, loan approval, and contract signing), including but not limited to photocopies of ID cards, income certificates, loan application forms, bank statements, and contract texts.

[0102] Document images refer to electronic images of financial documents captured by devices such as scanners, document scanners, and mobile phone cameras; they are usually image files.

[0103] One possible approach is to acquire images of the financial documents to be processed. Understandably, the sources of these images can be scanned copies uploaded by a scanner, certificate images collected by a self-service terminal, photo files uploaded by a mobile terminal, or historical contract images, income certificate images, bank statement images, or identity document images imported in batches from a business system.

[0104] After acquiring the document image, it undergoes preprocessing, including operations such as image denoising, tilt correction, and resolution normalization. Preprocessing effectively eliminates noise, angular distortion, and size differences introduced by shooting or scanning, improving the accuracy of subsequent character recognition. For example, for tilted ID photos, automatic rotation correction keeps them level; for blurry images, image enhancement algorithms are applied to improve clarity.

[0105] After preprocessing, OCR is used to detect and recognize text regions in the image. Specifically, the text lines and character positions in the image are located, and a deep learning model maps the image features of each character to the corresponding character code, while outputting a confidence score for each character. This confidence score represents the model's certainty about the recognition result. Finally, all the recognized characters are combined into coherent text data according to their positional order, and stored along with the coordinate information and confidence score information of each character for use in subsequent steps.

[0106] S102. Perform image feature analysis on the document images and semantic parsing on the text data.

[0107] Image feature analysis refers to extracting non-textual visual structural information from document images, such as the shape and position of a seal, the trajectory of a signature, the row and column layout of a table, the boundary coordinates of a text box, and the checked state of a checkbox.

[0108] Layout structure information refers to the spatial relationship of various information areas (such as title bar, table body, signature area) in a document image. For example, the "Loan Amount" field is usually located in a specific row and column of a table, with the corresponding amount value immediately to its right or below.

[0109] One possible implementation involves parallel execution of two processing chains: image feature analysis and text semantic parsing. In the image feature analysis chain, layout analysis is performed on the document image to identify structured regions such as tables, text boxes, seal areas, and signature bars. For example, for a scanned loan contract, the starting row of the table and the boundary coordinates of each cell are automatically detected. Simultaneously, a seal detection model locates the position of the red circular seal and extracts its texture features for subsequent authenticity verification. Furthermore, the continuity and pressure distribution of the handwriting in the signature area are detected to provide evidence for determining whether the signature is forged.

[0110] The text data output by OCR is processed using a natural language processing model. Specifically, field-level segmentation is performed, dividing continuous text into several independent fields or phrases based on line breaks, spaces, punctuation, etc. Then, named entity recognition technology is used to label key business entities in the text; for example, "100000" is labeled as an amount entity, and "December 31, 2025" is labeled as a date entity. Simultaneously, pattern matching or deep learning models are used to identify semantic anchors. For example, after finding the anchor "Loan Amount:", the numeric content immediately to its right is determined as the specific value of the "Loan Amount" field, and a mapping relationship between field names and field values ​​is established.

[0111] The image feature analysis results are correlated and aligned with the semantic analysis results. For example, semantic analysis might find that the text value corresponding to "loan amount" is "500000," while image feature analysis provides the actual coordinates of this text value within the document image and whether that region belongs to the "amount" column of the table. The two complement each other, providing a foundation for subsequent confidence assessment and location verification.

[0112] S103. Based on the image feature analysis results and semantic parsing results, determine the confidence score of the text data.

[0113] The confidence score refers to the quantitative evaluation value of the credibility of each target element and the overall text data. It is usually expressed as a percentage or decimal and is used to reflect the accuracy of character recognition and the rationality of the layout.

[0114] Image feature analysis results include information such as the image coordinate regions of each target element in the document image, the table row and column structure, and the location of seals and signatures.

[0115] The semantic parsing results include the field type identification results in the text data, the correspondence between field names and field values, and the semantic rationality score of each target element.

[0116] One possible approach is to segment the text data according to target elements after completing image feature analysis and semantic parsing. For example, in a loan application form, key information units such as "applicant's name," "ID number," "loan amount," and "monthly income" can be extracted independently. For each target element, its corresponding layout matching score is obtained from the image feature analysis results. The layout matching score reflects the degree to which the actual position of the element in the document image matches the position of the standard template.

[0117] The character confidence score of each target element is obtained from the character recognition results, which is the probability assessment of the correctness of the character sequence by the recognition model. In addition, the semantic reasonableness score can be extracted from the semantic parsing results to determine whether the field value conforms to common business sense (e.g., monthly income should not be negative).

[0118] After obtaining the scores from the above multiple dimensions, the pre-configured weight parameters are called. Different field types can be configured with different weight combinations. For example, for core fields such as "loan amount," the weight of character confidence may be higher; while for auxiliary fields such as "remarks," the weight of layout matching may be higher. The character confidence, layout matching sub-score, and semantic reasonableness score of each target element are multiplied by their corresponding weight coefficients, and then the weighted results are summed to obtain the comprehensive confidence sub-score of the target element.

[0119] After calculating sub-scores for all target elements, the sub-scores are aggregated, for example, using a weighted average to obtain the overall confidence score for the entire text data. For instance, if a document contains five target elements, and each element is of equal importance, the arithmetic mean of the five sub-scores is taken; if some elements (such as contract amount) are more critical, they can be assigned higher weights before averaging. The final output confidence score serves as the trigger condition for subsequent error correction processes; the natural language processing model's error correction mechanism will only be activated when the score falls below a preset threshold.

[0120] S104. When the confidence score is lower than the preset threshold, the text data is corrected using a natural language processing model.

[0121] Natural language processing models refer to language models trained based on deep learning technology, which can understand the contextual semantics, syntactic structure, and business logic of text, and are used to detect and correct fields with recognition errors.

[0122] Error correction refers to the process of inferring the correct value of a target element with low confidence using semantic context and business rules, and replacing the original erroneous content.

[0123] One possible implementation involves comparing the calculated overall score with a preset threshold. This preset threshold can be flexibly configured based on the business scenario. For example, in a loan approval scenario, where the accuracy requirements for fields such as amount and date are extremely high, the threshold can be set to 90%; while in a general information registration scenario, the threshold can be appropriately reduced to 80%. If the overall confidence score is not lower than the preset threshold, the current text data is considered to be of acceptable quality, and the error correction step is skipped to proceed to subsequent fusion processing; if the score is lower than the preset threshold, the error correction process is triggered.

[0124] After error correction is triggered, specific target elements with confidence scores below a threshold are selected from the text data. For example, the overall document score may be lowered because the confidence scores of certain fields (such as ID numbers or loan amounts) are low. These low-confidence target elements are extracted one by one. For each low-confidence target element, its surrounding context text (such as the field names before and after it, adjacent characters, or table header content) is input into a pre-trained natural language processing model.

[0125] Natural language processing models utilize contextual semantic information to re-analyze and infer low-confidence fields. Combining financial business logic rules (such as numerical range constraints, date format specifications, and ID number verification algorithms) with semantic relationships between fields (e.g., "loan amount" should not exceed ten times "monthly income"), the model infers the most likely correct value for the target element. For example, if OCR misinterprets "100000" as "10000", the model, through context, discovers that the field is "annual income" and the adjacent field "monthly income" is "8000", thus inferring that the annual income should be around 96000, and correcting "10000" to a more reasonable value. After correction, the corrected text data is output for subsequent fusion to generate structured data.

[0126] S105. The image feature analysis results are fused with the analysis results obtained by re-semantically analyzing the corrected text data to generate structured data.

[0127] Among them, the image feature analysis results refer to the visual structural information extracted from the document image, including the position and shape of the seal, the outline of the signature handwriting, the row and column coordinates of the table, the bounding boxes of each text area, and the spatial layout relationship between fields.

[0128] Structured data refers to document information organized according to a predefined data model. Each field has clear attributes such as type, value, location coordinates, and confidence level, which facilitates subsequent automated verification and business processing.

[0129] Understandably, text data only contains character sequences and basic semantic information, and has not yet been associated with visual structures in the document image (such as tables, seals, and signature locations). Therefore, it is necessary to re-parse the corrected text data semantically to ensure that the semantic type and boundaries of each field are accurately identified. The reason for re-parsening is that the original semantic parsing was based on potentially erroneous text, and field segmentation and anchor point positioning may be inaccurate; the corrected text is more reliable, and re-parsening can obtain a more accurate field-value mapping relationship.

[0130] The previously extracted image feature analysis results are read from the memory, including the image coordinates of each text region (such as the coordinates of the four corner points of a rectangle), whether the region belongs to the table body, and whether it is located inside the signature bar. Subsequently, the fields and their values ​​obtained from the re-semantic parsing (e.g., the "loan amount" field value is "500000") are matched and aligned with the corresponding coordinate regions in the image feature analysis results.

[0131] The matching can be based on the position index of the field value in the text stream or the image coordinates of the field name anchor point. Through alignment, it becomes clear where the value "loan amount" appears in the document image and whether that position matches the amount field in the standard template.

[0132] After alignment, the information from the two sources is fused to generate structured data. Specifically, each target element is treated as a row or an object, containing a field name, field value, image coordinate region of the field value, field layout type (such as header, data cell, signature bar), and additional attributes obtained from image feature analysis (such as seal overlap, handwriting continuity score, etc.). The fused structured data not only preserves the business meaning of the text but also retains the document's physical layout and visual evidence, providing support for subsequent knowledge graph contradiction detection and hierarchical correction.

[0133] S106. Based on financial knowledge graphs, perform semantic contradiction detection on structured data to determine contradictory fields and their contradiction risk levels.

[0134] Among them, financial knowledge graph refers to a pre-constructed semantic network that stores financial business knowledge in a graph structure. It includes financial business entities (such as customers, loan products, and contract terms), entity attributes (such as customer age and loan amount limit), and the relationship and constraint rules between entities (such as "monthly repayment amount shall not exceed 50% of monthly income").

[0135] Conflicting fields refer to structured data fields that are found to conflict with constraint rules in the knowledge graph during consistency checks.

[0136] The type of contradiction is a classification of the nature of the contradiction, such as numerical contradictions (e.g., inconsistent amount calculations), timeliness contradictions (e.g., dates exceeding the expiration date), identity contradictions (e.g., name mismatch), and business rule contradictions (e.g., violations of regulatory red lines).

[0137] The conflict risk level is a severity label assigned to each conflict according to a preset risk level mapping rule, which is used for subsequent classification and correction decisions.

[0138] One possible implementation involves loading a pre-built financial knowledge graph. This knowledge graph is pre-constructed using expert rules, regulatory documents, and historical data, and can be dynamically updated based on business changes. The knowledge graph stores common entity relationships in financial transactions, such as constraints like "loan amount cannot exceed ten times monthly income" and "contract effective date should be earlier than the expiration date of ID document validity." Each field in the structured data is parsed into a target entity or entity attribute in the knowledge graph. For example, "Loan amount: 500,000" in the structured data is mapped to the "Loan amount" attribute node in the knowledge graph; "Borrower name: Zhang San" is mapped to the "Customer" entity.

[0139] After mapping is complete, a consistency verification engine is activated. This engine traverses the relationships and constraints related to the mapped entities in the knowledge graph, verifying each field value in the structured data to ensure it meets the rule requirements. For example, for the "loan amount" field, it checks for the constraint "loan amount ≤ monthly income × 10" and automatically locates the value of the "monthly income" field in the structured data for comparison. If the rule is met, the verification passes; otherwise, it is determined to be a "numerical contradiction." Similarly, time-related rules (such as the order of ID card validity period and contract date), identity rules (such as the consistency of borrower and mortgagor names), and regulatory rules (such as interest rate caps) are verified one by one. Each time a contradiction is found, the rule that triggered the contradiction, the fields involved, and the specific conflicting value are recorded.

[0140] After all verifications are completed, the risk level of each contradiction is determined based on the detected contradiction type and degree of conflict, combined with preset risk level mapping rules. For example, contradictions that violate regulatory rules (such as loan interest rates exceeding the legal limit) are automatically classified as high-risk; minor numerical deviations caused by OCR misidentification (such as a missing zero at the end of an amount that can be corrected through business context) are classified as low-risk. The final determined contradiction fields, contradiction types, and corresponding risk levels are output for use in subsequent classification and correction steps.

[0141] S107. Based on the corresponding conflict risk level, correct the conflict field to obtain the corrected structured data and output it.

[0142] Among them, the correction value refers to the correct value generated according to the correction rules or manual feedback, which is used to replace the original erroneous value in the contradictory field.

[0143] One possible implementation involves adopting differentiated correction strategies based on different risk levels. Specifically, the list of contradictory fields and their corresponding risk level identifiers output from previous steps are read. For contradictions determined to be low-risk, their error type is considered clear, the correction rules are mature, and they will not introduce new risks, so they directly enter the automatic correction branch. For example, for a numerical contradiction triggered by OCR misinterpreting "100000" as "10000", the numerical error correction rules in the preset rule base are called, and a corrected value is automatically generated based on the context (such as the capitalized amount in a loan contract or adjacent monthly repayment amounts). This corrected value then directly replaces the erroneous value in the contradictory field, completing the correction.

[0144] For contradictions deemed high-risk, automatic correction is deemed potentially uncertain or subject to compliance risks. Therefore, instead of automatic replacement, a detailed warning report is generated. This report clearly lists the original value of the contradictory field, the suggested correction value provided by the rule base or knowledge graph, a detailed description of the contradiction type (e.g., "contract date is later than ID card expiration"), and correction suggestions for reviewers. This warning report is then pushed to a human review terminal, awaiting feedback from reviewers. For example, reviewers can view the contradiction details on the terminal, choose to adopt the suggested correction value, manually enter a new correction value, or confirm that the contradictory field does not need modification (e.g., exceptions allowed by rules in special business scenarios). The system receives confirmation or modification instructions from the review terminal and updates the structured data based on the final correction value in the instructions.

[0145] After correcting all contradictory fields (whether automatically or manually), the updated structured data is integrated to form complete, contradictory-free corrected structured data. This data is organized according to a predefined business data format and includes the correct values ​​of all fields, coordinate information, and records of the correction (such as the source and time of the correction). Finally, the corrected structured data is output to downstream business processes, such as being pushed to a loan approval system, archived to an image platform, or used to generate compliance reports.

[0146] The document processing method based on multimodal feature fusion provided in this application involves: acquiring a document image of a financial document to be processed and performing character recognition on the document image to obtain text data; performing image feature analysis on the document image and semantic parsing on the text data; determining a confidence score for the text data based on the image feature analysis results and the semantic parsing results; correcting errors in the text data using a natural language processing model when the confidence score is lower than a preset threshold; fusing the image feature analysis results with the parsing results obtained from re-semantically parsing the corrected text data to generate structured data; performing semantic contradiction detection on the structured data based on a financial knowledge graph to determine contradictory fields and their contradiction risk levels; and correcting the contradictory fields according to the corresponding contradiction risk levels to obtain and output the corrected structured data. In this embodiment, a continuous processing chain for financial document processing is constructed by sequentially linking character recognition, image feature analysis, semantic parsing, confidence control, natural language error correction, multimodal structured fusion, financial knowledge graph contradiction detection, and risk classification correction. This chain can improve the accuracy of information extraction, the completeness of logical verification, and the targeting of risk identification and correction under conditions of complex layouts, low-quality images, cross-field conflicts, and dynamic business rules. It also reduces the probability of erroneous data entering the approval, archiving, and risk control processes, thereby enhancing the stability, traceability, and business adaptability of the financial document processing process.

[0147] Figure 2 A flowchart illustrating the document processing method based on multimodal feature fusion provided in this application. Figure 2 ,like Figure 2 As shown, in this embodiment... Figure 1 Based on the examples, a document processing method based on multimodal feature fusion is described in detail. This method includes:

[0148] S201. Calculate the layout matching degree based on the layout structure information in the semantic parsing results and the layout structure information in the image feature analysis results.

[0149] Among them, layout matching degree is a numerical indicator used to quantify the degree of similarity between the actual layout of a document and the expected standard layout. The higher the matching degree, the more consistent the field position in the document is with the standard position in the preset template or business specification, thus indirectly proving that the recognition result of that field is more reliable.

[0150] One possible approach is to extract the field types corresponding to each target element in the text data from the semantic parsing results, such as "loan amount," "annual interest rate," and "contract date." Simultaneously, the actual image coordinate regions of these target elements within the document image are obtained from the image feature analysis results; that is, the rectangular frame position occupied by each field value in the image. For example, for the "loan amount" field, the coordinates of its text value "500000" at the top left and bottom right corners of the image.

[0151] Invokes a pre-defined financial document template. The template predefines the standard coordinate range in which each field should appear for the currently processed document type (such as a standard mortgage contract). For example, the template specifies that "Loan Amount" should be located in the third row and second column of the table, with a coordinate range of (200, 300) to (400, 350).

[0152] The actual image coordinate regions are compared one by one with the corresponding standard coordinate regions in the template. For each target element, the spatial difference between its actual coordinate region and the standard coordinate region is calculated. If the actual region and the standard region completely overlap or the deviation is minimal, it indicates that the position of the field in the document highly matches the template, and the layout matching degree is high; conversely, if the deviation is large, it indicates that the document is tilted, scaled, or the field recognition position is incorrect. The deviation values ​​of all target elements are converted into layout matching degree sub-scores between 0 and 100 according to preset mapping rules (e.g., the smaller the deviation, the higher the score), and aggregated (e.g., taking the average or weighted average) to obtain the layout matching degree of the entire document.

[0153] S202. Based on the character confidence score and layout matching degree output during character recognition, determine the confidence score of the text data.

[0154] One possible implementation involves obtaining the character confidence score for each target element from the character recognition module. For a target element containing multiple characters (such as "loan amount" composed of multiple digits), the average of these character confidence scores is taken as the overall character confidence score for that element. Simultaneously, the layout matching score corresponding to the target element is obtained.

[0155] For each target element, pre-configured weight parameters are called. Different field types can be configured with different weights; for example, the character confidence score weight for the amount field is set to 0.7, and the layout matching score weight is set to 0.3, while the signature field places more emphasis on layout matching. The character confidence score and layout matching score sub-scores are multiplied by their respective weights and then summed to obtain the overall confidence score sub-score for that element. This ensures that the final score considers both the accuracy of character recognition and the reasonableness of the field's position in the document.

[0156] After calculating the overall confidence score for all target elements, the sub-scores are summarized. For example, a weighted average method can be used, where each sub-score is weighted according to the business importance of that field (with preset weights), summed, and then divided by the total weight. The final value is the confidence score for the entire text data.

[0157] By correlating the layout structure information from semantic parsing results with that from image feature analysis results, and then combining this with the character confidence score output during character recognition to comprehensively evaluate text data, the system avoids relying solely on character recognition results while ignoring layout consistency. This improves the accuracy of identifying fields such as amount, date, document number, and signature information in financial documents, reduces the impact of field misalignment or incorrect attribution on subsequent semantic parsing and structured processing, and enhances the system's adaptability to low-quality images, complex tables, and documents with overlapping seals.

[0158] S203. When the confidence score is lower than the preset threshold, extract the target elements in the text data whose confidence scores are lower than the preset threshold.

[0159] One possible implementation involves calculating the overall confidence score for the entire text data and then comparing it to a preset threshold. If the overall score is higher than or equal to the threshold, the document recognition quality is considered acceptable, and the process proceeds to subsequent fusion processing. Conversely, if the overall score is lower than the threshold, it indicates the presence of one or more target elements with poor recognition quality in the document, requiring error correction to be initiated.

[0160] The algorithm iterates through each target element in the text data, obtaining a comprehensive confidence sub-score for each element. The sub-scores are then compared to a preset threshold (or an independent threshold at the field level), filtering out target elements with sub-scores below the threshold. For example, in a loan contract, the sub-score for "Loan Amount" is 92%, the sub-score for "ID Number" is 78%, and the sub-score for "Contract Date" is 95%. If the preset threshold is 85%, only the "ID Number" target element will be extracted for error correction.

[0161] S204. Locate the semantic anchor point of the target element in the document. The semantic anchor point includes at least one of the field name of the target element, adjacent target elements, or table headers.

[0162] S205. Based on the contextual information of semantic anchors and combined with financial business logic rules, infer the correct value of the target element.

[0163] Semantic anchors refer to textual features in a document that have clear identifying meanings and can help locate the exact position and meaning of target elements. Semantic anchors include the field name corresponding to the target element (such as "Loan Amount:"), other target elements adjacent to the target element (such as the value next to the "Monthly Income" field), and table headers in tables (such as the column title of a column in a table).

[0164] One possible approach is to locate semantic anchors within the document for extracted low-confidence target elements. The location method varies depending on the document structure: for key-value pairs (e.g., "Applicant: Zhang San"), search for the nearest field name to the left or above the target element as a semantic anchor, for example, finding the text "Applicant:". For data cells in a table, search upwards to the column header cell (e.g., "Monthly Income (Yuan)") as an anchor, while also searching to the left for the row header label (e.g., "Borrower Information"). Furthermore, adjacent target elements can be used as auxiliary anchors; for example, when the "Loan Amount" field has low confidence, adjacent fields with higher confidence, such as "Loan Term" or "Annual Interest Rate," can be used to construct richer context.

[0165] After locating the semantic anchor, the text content, positional relationships, and adjacent field values ​​identified with high confidence are extracted from the anchor and its surroundings. For example, for a low-confidence monetary value, it is known that there is a field name "Loan Amount:" on the left, the header "Amount (Ten Thousand Yuan)" above, and the adjacent field "Loan Term" on the right is "36 Months". This contextual information is then input into the natural language processing and rule-based reasoning engine.

[0166] The inference engine determines the business type that the target element should meet based on the anchor point (e.g., an amount field, where the unit might be ten thousand yuan or yuan). The engine then loads financial business logic rules related to that business type. For example, the rule base contains constraints such as "the loan amount shall not exceed 10 times the applicant's annual income" and "the monthly repayment amount shall not exceed 50% of the monthly income." If the "monthly income" of 8,000 yuan has already been obtained from other high-confidence fields, then for the low-confidence "loan amount," the engine can infer its reasonable range, for example, not exceeding 8,000 × 10 × 12 = 960,000 yuan. At the same time, the engine also combines format rules (e.g., the amount should not contain letters, and there should be a maximum of two decimal places) and common typo patterns (e.g., missing a zero, or adding an extra separator) to generate candidate correction values.

[0167] The inference engine ultimately outputs one or more candidate correct values, along with a confidence score for each candidate value (different from the original OCR confidence score, but an inference confidence score based on the degree of rule satisfaction). If there is only one candidate value and the inference confidence score is higher than a preset threshold, that value is taken as the "inferred correct value"; if there are multiple candidates, the one with the highest rule matching degree and the best consistency with the context is selected; if there are no candidates, the target element is marked as not being automatically inferred and awaits subsequent text replacement or manual review processes.

[0168] S206. Replace the incorrect value in the target element with the inferred correct value to obtain the corrected text data.

[0169] One possible implementation involves locating the precise position of the target element within the original text data during the replacement process. The character recognition results preserve the index or coordinate range of each character, allowing for accurate identification of the start and end positions of the character sequence to be replaced.

[0170] After location is complete, the original erroneous character sequence is deleted entirely and replaced with the inferred correct value. During replacement, the order of the original target element in the text stream is preserved, while the element's length information is updated. For example, if the original OCR identifies "loan amount" as "10000" but the inferred correct value is "100000", then the three characters "10000" are replaced with the five characters "100000".

[0171] During the replacement process, logs are recorded for each replacement operation, including the original error value, the inferred correct value, and the basis for the replacement (such as which business rule or semantic anchor was used), for subsequent auditing or traceability. Finally, all corrected fields are recombined into new text data in their original order. In the new text data, fields that were originally low-confidence have been covered by reliable correct values, while fields that were originally high-confidence remain unchanged.

[0172] S207. The image feature analysis results are fused with the analysis results obtained by re-semantically analyzing the corrected text data to generate structured data.

[0173] Step S207 is similar to step S105 above, and will not be described again here.

[0174] The process involves locating the object to be corrected from low-confidence elements, restoring its role and position within the document, and then jointly constraining the contextual semantics and financial business rules into the candidate value space. This ensures that the model outputs the correct value consistent with the document structure and business logic. The correction process does not rely solely on character similarity but also utilizes field relationships and business rules, thus improving the accuracy and stability of financial document text correction, reducing the probability of erroneous values ​​directly entering subsequent structured processing stages, and lowering the cost of manual review.

[0175] S208. For fields in structured data, find the target entity and entity attributes corresponding to each field from the financial knowledge graph.

[0176] Financial knowledge graphs are pre-built semantic networks that store financial business knowledge in a graph structure. This includes financial business entities, entity attributes, relationships between entities, and constraint rules.

[0177] One possible implementation involves, after obtaining the structured data, iterating through each field to retrieve its name (e.g., "Loan Amount") and value (e.g., "500000"). A pre-built financial knowledge graph is then loaded, and the field name is used as the query key to search the entity attribute index within the knowledge graph. For example, the search could employ exact matching (e.g., field name and attribute name are exactly the same) or semantic similarity matching (e.g., "Total Loan Amount" matches the "Loan Amount" attribute). If a match is successful, the target entity to which the attribute belongs can be located. For instance, the attribute "Loan Amount" belongs to the "Loan Product" entity, and the attribute "Borrower Name" belongs to the "Customer" entity.

[0178] For each field, record the target entity type and the specific attribute corresponding to that field. If a field cannot be directly matched to an attribute name, use the relationships in the knowledge graph for indirect lookup. For example, the field "repayment account" may not be a direct attribute of any entity, but it can be found by associating the "repayment account" entity with the "loan contract" entity. After the lookup is complete, save the correspondence between each field and the target entity and entity attribute as a mapping table for subsequent consistency checks.

[0179] S209. Based on the relationships and constraint rules in the financial knowledge graph, perform consistency verification on the target entity and entity attributes.

[0180] S210. If the verification result is inconsistent, locate the corresponding contradictory field and contradiction type in the structured data according to the inconsistent entity attributes, and determine the risk level of the contradiction according to the preset risk level mapping rules.

[0181] Understandably, after mapping fields to the knowledge graph, all mapped records are traversed, and consistency checks are performed on each field (or combination of fields). For example, the check process is divided into single-field checks and cross-field checks. Single-field checks verify whether the field value itself meets the basic constraints of its attribute, such as whether "loan amount" is non-negative or whether "ID number" conforms to the checksum algorithm. Cross-field checks utilize the relationships in the knowledge graph. For example, if there is a constraint rule between the "loan amount" attribute under the "loan contract" entity and the "monthly income" attribute under the "customer" entity ("amount ≤ monthly income × 10"), the system automatically retrieves the corresponding two field values ​​in the structured data and calculates whether this inequality is satisfied. If satisfied, the check passes; otherwise, the record is considered inconsistent.

[0182] Once the verification finds an inconsistency, the specific entity attribute causing the inconsistency is located, and the original field in the structured data corresponding to that attribute is found in reverse through the mapping table, and that field is marked as a contradictory field.

[0183] At the same time, the type of conflict is determined based on the content of the violated rules: violations of numerical calculation rules are classified as numerical conflicts, violations of date order rules are classified as timeliness conflicts, violations of identity matching rules are classified as identity conflicts, and violations of regulatory or internal risk control indicators are classified as business rule conflicts.

[0184] The system invokes preset risk level mapping rules to assign a risk rating to the conflict. Specifically, the mapping rules comprehensively consider three factors: first, the inherent risk of the conflict type (e.g., regulatory conflicts are directly classified as high risk); second, the degree of deviation (e.g., the severity of a loan amount exceeding the limit by 10% is different from exceeding it by 100%); and third, the business importance of the field (e.g., deviations in the core amount field are more severe than those in the remarks field). Based on preset thresholds or a decision table, the final risk level (low, medium, high) is obtained.

[0185] By linking field mapping, graph consistency verification, and risk grading, structured data can not only be verified at the literal level but also uncover hidden contradictions at the business semantic level. This effectively identifies inconsistencies across fields, entities, and documents, reducing the probability of erroneous data entering subsequent credit granting, contract, and archiving stages, and enhancing the automated review capabilities of financial document processing systems.

[0186] S211. If the risk level is low, the preset rule base is called, the corresponding correction rule is matched according to the contradiction type, the correction value is obtained, and the error value in the contradiction field is replaced with the correction value to obtain the corrected structured data.

[0187] Among them, the correction rules refer to the preset correction logic for specific contradiction types. For example, the monetary error correction rule can automatically complete the correction based on the reasonableness of adjacent fields (such as uppercase amounts) or the number of digits.

[0188] One possible implementation is to automatically enter the rule base correction branch for contradictory fields marked as low-risk. Specifically, a preset rule base is invoked, and the corresponding correction rule is matched in the rule base according to the contradiction type. For example, if the contradiction type is "numerical error in amount," then the amount correction rule is matched; if it is "date format error," then the date standardization rule is matched.

[0189] After a successful match, the original value of the conflicting field and its auxiliary context (such as uppercase amounts in the same line, adjacent date fields, etc.) are extracted. The original value and context are then input into the correction rule to perform the correction calculation. For example, the correction rule checks whether the number of digits in the original value conforms to the common range of the business scenario (such as the amount in a loan contract, which is usually between tens of thousands and tens of millions of yuan). If the number of digits is obviously missing (such as "10000" should be "100000"), zeros are automatically added to the end; if there are illegal characters (such as the letter O in "1OO000"), they are inferred and replaced based on adjacent numbers or uppercase amounts.

[0190] After obtaining the corrected values, replace all erroneous values ​​in the contradictory fields with the corrected values, and update the record for that field in the structured data (including the value and confidence level). If a structured dataset contains multiple low-risk contradictory fields, process them one by one. After all replacements are complete, output the corrected structured data.

[0191] S212. If the risk level is high risk, generate an early warning report containing the original value of the contradiction field, the suggested correction value, the contradiction type description, and the correction suggestion, and push the early warning report to the review terminal.

[0192] One possible implementation involves not performing any automatic correction operations on contradictory fields marked as high-risk, but instead redirecting to the warning report generation branch. Specifically, the original value of the contradictory field (i.e., the currently identified value violating the rules) is extracted from the structured data. Based on the specific constraint rules violated in the consistency check, combined with standard values ​​from the knowledge graph or derivation logic from the rule base, one or more suggested correction values ​​are generated. For example, for the contradiction of "contract date later than ID card expiration date," a reasonable date (e.g., the day before the expiration date) is calculated backward from the ID card's expiration date as the suggested value.

[0193] Generates conflict type descriptions and correction suggestions. The conflict type descriptions use templated natural language, automatically filling in key information such as conflict field names, original values, and rule thresholds, making it easy for reviewers to understand. Correction suggestions provide specific operational guidance based on the conflict type, such as "Please adjust the interest rate to the recommended range according to the latest regulatory requirements." This information is integrated into a complete early warning report, which can also include screenshots or highlighted coordinates of the conflict fields' locations in the original document image, facilitating quick location by reviewers.

[0194] The generated warning report is pushed to the designated review terminal. This can be done by dispatching a task through the internal workflow system or by sending it to the manual review interface in real time via a message queue. Upon receiving the report, the review terminal will highlight the conflict details on its interface and provide interactive options such as "Adopt suggested value," "Manually modify," and "Ignore and note the reason." At this point, the handling of high-risk conflicts has been transferred from the automated process to the manual process, awaiting further confirmation and correction by the reviewers.

[0195] Differentiated processing strategies are adopted for different risk levels. Specifically, contradictory fields that can be resolved through rules are directly replaced, while high-risk contradictions that may affect business conclusions retain a review process. Through the collaboration of rule base matching, correction value generation, early warning report organization, and terminal push, the accuracy and consistency of contradictory field corrections can be improved, reducing the workload of manual item-by-item verification. Specifically, for low-risk scenarios, the efficiency of structured data entry can be improved; for high-risk scenarios, anomaly tracking and audit traceability can be enhanced, thereby improving the stability of the financial document processing system.

[0196] In some embodiments, when the risk level is high, a confirmed correction report is received from the audit terminal;

[0197] Based on the correction values ​​in the correction report, the contradictory fields are corrected to obtain the corrected structured data, which is then output.

[0198] The review terminal can be a manual review workstation within a financial institution, a business terminal with signature and approval functions, or a mobile review device with review authority.

[0199] The confirmed correction report refers to the review result generated by the reviewers after combining the original document images, business rules and manual judgment for high-risk contradictory fields. The correction report should at least include the contradictory field identifier, correction value, correction basis and confirmation status.

[0200] Understandably, after determining the risk level to be high, an alert message containing the original value, suggested correction value, and a description of the contradiction type is pushed to the review terminal. Reviewers confirm, modify, or reject the suggested correction value on the terminal and send the confirmed correction report back to the backend processing module. The backend processing module parses the correction report, locates the target record in the structured data according to the field identifiers, and overwrites the original value with the correction value, forming the corrected structured data.

[0201] By introducing a closed-loop manual verification process in high-risk scenarios, high-risk fields are not automatically rewritten directly. Instead, controlled corrections are made based on the verified correction reports returned by the auditing terminal, thus ensuring the accuracy of error correction. Since the corrections are based on manually reviewed correction values, the probability of high-risk fields being erroneously replaced by automatic means is effectively reduced, and the final structured data output is more suitable for financial business scenarios such as credit approval, contract archiving, and risk tracking.

[0202] The document processing method based on multimodal feature fusion provided in this application calculates the comprehensive confidence score of each target element based on the layout structure information of the document image and the character confidence score output by character recognition. When the score is low, the semantic anchor point of the target element (such as field name, adjacent content, or table header) is located, and the correct value is inferred by combining financial business logic rules to replace the erroneous text, achieving accurate error correction from "character level" to "semantic level". Then, the corrected text is re-fused with the image feature analysis results to generate structured data. The structured data is then checked for consistency based on a pre-built financial knowledge graph (including entities, attributes, and constraint rules). If contradictions are found, the conflicting fields are located, the contradiction type is determined, and the risk level is classified. Low-risk contradictions are automatically corrected by calling the rule base, and high-risk contradictions generate warning reports and push them for manual review. This solves the problems of difficult elimination of OCR recognition errors and missed detection of cross-field logical contradictions in the prior art, improves the accuracy of financial document information extraction, effectively identifies potential risks, and reduces the burden of manual review.

[0203] Figure 3 A flowchart illustrating the document processing method based on multimodal feature fusion provided in this application. Figure 3 ,like Figure 3 As shown, in this embodiment... Figure 1 Based on the embodiments, a possible implementation method for determining the confidence score of text data based on image feature analysis results and semantic parsing results is described in detail. The method includes:

[0204] S301. Based on the semantic parsing results, determine the fields corresponding to each target element in the text data.

[0205] As is understandable, semantic parsing results refer to the output obtained after analyzing text data through natural language processing techniques, such as word segmentation, part-of-speech tagging, named entity recognition (e.g., amount, date, name), and candidate correspondence between field names and values.

[0206] One possible implementation is to extract all identified candidate field names (anchors) and candidate target elements (values) from the semantic parsing results; traverse the entire text data and, according to the sequence labeling results in natural language processing, segment the continuous text into several candidate field name-value pairs.

[0207] For tables, field names are not directly adjacent to the target element, but appear in the table header or at the beginning of the row. Dependency analysis and table structure recognition results from semantic parsing are used to associate information scattered in different locations. For example, the first column header of the table is "Item," and the second column header is "Amount"; one row in the table contains "Loan Principal" and "500,000." It is inferred that the target element "500,000" corresponds to the field "Loan Principal" (through the combination of the item name at the beginning of the row and the "Amount" in the column header).

[0208] If the same target element may correspond to multiple candidate fields (such as the value "100000" which could be "loan amount" or "annual income"), the best candidate is selected based on the context semantics, the distance between field names, and whether the field has already appeared.

[0209] Each successfully identified target element is labeled with a specific field, forming a field-value mapping table.

[0210] It cannot be determined whether the target element of the corresponding field is marked as "uncategorized" and will be handled further.

[0211] S302. Extract the image coordinate regions of each target element in the document image from the image feature analysis results.

[0212] Understandably, the image feature analysis results contain information on all detected text regions in the document image. Each text region has corresponding text content (i.e., the character sequence recognized by OCR) and its coordinate bounding box in the image. The previous step determined the text value of each target element and its position index in the text data. The next step is to establish the association between the target element and the image coordinate region.

[0213] One possible implementation involves iterating through each target element to be extracted, obtaining its text content and the starting and ending character positions of that text in the complete text data; since each detection box in the image feature analysis results also records the text content identified within the box and its approximate position in the original text (usually assigned after sorting by a layout analysis algorithm); performing precise string matching or fuzzy matching between the text content of the target element and the text within the detection box; if a complete match or similarity exceeds a preset threshold (such as 95%), then the coordinate region of the detection box is used as the image coordinate region of the target element.

[0214] S303. Based on the preset financial document template, determine the preset coordinate area corresponding to each target element.

[0215] The preset coordinate area refers to the standard position range predefined for each field in the template, usually represented by a rectangular bounding box.

[0216] One possible implementation involves loading a preset financial document template corresponding to the currently processed financial document type; iterating through all target elements whose fields were determined in the previous steps, and obtaining the field name for each target element, such as "Loan Amount," "Borrower Name," and "Contract Date." For each field name, a precise match or semantic match is performed in the preset template (e.g., "Total Loan Amount" matches "Loan Amount" in the template); upon successful matching, the preset coordinate region corresponding to that field is read from the template and stored in pixel coordinates or relative percentages.

[0217] If a target element cannot find a corresponding preset coordinate region in the template (e.g., the template version is older and the field is missing), the default strategy is used. Specifically, it is estimated by interpolation based on the average position of the field in other similar templates, or by using the preset coordinates of adjacent known fields.

[0218] For fields that are completely uncertain, mark them as "no preset area" and skip the position evaluation when calculating the layout matching degree.

[0219] S304. For each target element, calculate the distance deviation between the image coordinate region of the target element and the preset coordinate region.

[0220] Understandably, for each target element for which both image coordinates and preset coordinates have been obtained, distance deviation calculation is initiated. Specifically, key geometric parameters are extracted from the two regions: the center point coordinates, width, and height of the image coordinate region; and the center point coordinates, width, and height of the preset coordinate region. The distance deviation value is calculated using either the center point Euclidean distance or the boundary overlap deviation calculation method. The center point Euclidean distance calculates the straight-line distance between the centers of two rectangles; the boundary overlap calculates the ratio of the intersection area to the union area of ​​the two rectangles, with higher overlap resulting in smaller deviations. One method or a combination thereof can be selected depending on the document type and business requirements.

[0221] Specifically, the preset coordinate region is scaled proportionally to the actual size of the current document image; the horizontal and vertical deviations between the center point of the image coordinate region and the center point of the preset coordinate region are calculated; and the Euclidean distance is obtained using the Pythagorean theorem. For example, if the image center is (500, 300) and the preset center is (510, 310), then the horizontal deviation is 10 pixels, the vertical deviation is 10 pixels, and the Euclidean distance is 14.1 pixels.

[0222] Another possible implementation is to use the overlap method for calculation. Specifically, the area of ​​the intersection of the two rectangles is calculated and divided by the area of ​​the union to obtain a value between 0 and 1. This value is then subtracted from 1 to convert it into a deviation measure.

[0223] In some embodiments, a tolerance threshold can also be introduced, that is, when the deviation is less than the preset tolerance value, the deviation is directly output as 0; otherwise, the actual deviation minus the tolerance value is output.

[0224] S305. Convert the distance deviation value into a layout matching score according to the preset mapping rules.

[0225] S306. Calculate the layout matching degree based on the layout matching degree sub-score of each target element.

[0226] Understandably, after obtaining the distance deviation value for each target element, these deviation values ​​are converted into a unified, comparable layout matching sub-score. The conversion process relies on preset mapping rules. The mapping rules are, for example: when the deviation is zero, the sub-score is the highest score (e.g., a maximum score of 100); when the deviation is less than the "minor deviation tolerance threshold," a score close to full marks is given; when the deviation exceeds the "maximum acceptable deviation," the sub-score is zero; and when it is in between, the score decreases linearly as the deviation increases.

[0227] Specifically, the tolerance deviation value and maximum deviation value are read from the preset mapping rules. The tolerance deviation value indicates that the layout quality is not affected within the deviation range; the maximum deviation value indicates that the position is considered completely unacceptable if it exceeds the range. If the distance deviation value of the current target element is D, when D is less than or equal to the tolerance deviation, the sub-score is directly set to 100%; when D is greater than or equal to the maximum deviation, the sub-score is set to 0%; when D is between the two, the sub-score is calculated according to the linear interpolation formula, that is, the sub-score is equal to (maximum deviation D) divided by (maximum deviation tolerance deviation), and then multiplied by 100%.

[0228] After determining the layout matching sub-score for each target element, the individual scores are aggregated into a comprehensive score that represents the overall layout quality of the document; this is the layout matching score. The aggregation calculation commonly uses an arithmetic mean, which involves adding the sub-scores of all target elements and then dividing by the total number of target elements. For example, a loan contract contains 10 key fields, each with sub-scores of 95, 88, 92, 97, 85, 90, 96, 89, 93, and 91. The layout matching score would then be the average of these ten scores, approximately 91.6.

[0229] For documents where the importance of certain fields differs significantly (e.g., in a table, core data fields are more important than auxiliary description fields), a weighted average approach can be used. This involves pre-configuring a weight coefficient for each target element; for example, the weight of "Loan Amount" is set to 2, and the weight of "Remarks" is set to 0.5. The sum of the products of each sub-rating and its weight is calculated, and then divided by the sum of all weights to obtain the weighted average layout matching degree.

[0230] This matching accuracy calculation mechanism jointly verifies the field attribution in semantic parsing with the actual position in image features. This allows the system to identify situations where the text content is correct but the layout is misaligned, fields are crossed, or templates are inconsistent, reflecting these as lower layout matching accuracy. Therefore, the system can improve the reliability of financial document recognition results, enhance its adaptability to scenarios with tilted cameras, misaligned layouts, and complex tables, and provide stronger input for subsequent error correction, structured fusion, and semantic contradiction detection.

[0231] S307. Segment the text data according to the target element to obtain at least one target element to be scored.

[0232] S308. For each target element, obtain the character confidence score, the corresponding layout matching score, and the preset character confidence score weight and layout matching score weight.

[0233] Understandably, text data is broken down into independent target elements to facilitate subsequent refined scoring on a field-by-field basis. The segmentation is primarily based on field boundary information output from the semantic parsing stage. For example, the natural language processing model has already identified field name anchors in the text (such as "Loan Amount:") and their corresponding value regions, determining the start and end character positions of each target element based on the positional relationship between the anchors and values.

[0234] Specifically, based on the row and column relationships of the table, the text within each cell is treated as an independent target element. If a cell contains multiple subfields (such as province, city, and district in an address), it is further segmented based on phrase boundaries or punctuation marks from semantic parsing. During the segmentation process, meaningless spaces, line breaks, and purely decorative punctuation are removed to ensure that each target element contains only a sequence of characters with business meaning.

[0235] After the segmentation is completed, a list of target elements is output. Each element records its text content, the start and end indexes in the original text, and the corresponding field type (such as "loan amount").

[0236] For each target element to be scored, the character confidence score is extracted from the output of the optical character recognition module; the layout matching score corresponding to the target element is obtained from the previous steps; and based on the field type of the target element, the character confidence score weight and layout matching score weight corresponding to that field are read from a pre-configured weight table. The weight table can be obtained based on business expert experience or historical data statistical analysis. For example, in a loan contract, the character confidence score weight for the "Loan Amount" field can be set to 0.7, and the layout matching score weight to 0.3.

[0237] S309. Based on the character confidence score weight and the layout matching score weight, the character confidence score and the layout matching score sub-score are weighted and summed to obtain the comprehensive confidence score sub-score of the target element.

[0238] One possible implementation involves determining the character confidence weight and layout matching weight for the target element. These weights are typically pre-set by business experts based on the importance of the field and stored in a configuration table. For example, for the "Loan Amount" field in a loan contract, since numerical accuracy is crucial for risk control, its character confidence weight is set to 0.8, and its layout matching weight is set to 0.2.

[0239] Multiply the character confidence score by its weight to obtain the weighted character contribution value; simultaneously, multiply the layout matching score by its weight to obtain the weighted layout contribution value. Add these two weighted values ​​together to obtain the overall confidence score for the target element. For example, if the character confidence score is 0.92 and the character confidence weight is 0.7, and the layout matching score is 0.85 with a weight of 0.3, then the overall sub-score is 0.92 × 0.7 + 0.85 × 0.3 = 0.899.

[0240] Iterate through all target elements to be rated, calculating their respective comprehensive confidence sub-scores using the same method. After calculation, save the comprehensive confidence sub-score for each target element for use in subsequent steps.

[0241] S310. The confidence scores of the text data are obtained by weighted averaging of the comprehensive confidence scores of each target element.

[0242] Understandably, considering the varying importance of different fields in the business, a weighted average is typically used instead of a simple arithmetic average. For example, the weight value corresponding to each target element is obtained; then, a weighted average calculation is performed. Specifically, the overall confidence sub-score of each target element is multiplied by its corresponding weight, and all products are summed to obtain a weighted sum; simultaneously, the weights of all target elements are summed to obtain the total weight; finally, the weighted sum is divided by the total weight to obtain the confidence score of the text data.

[0243] The final confidence score is used for subsequent decisions. If the score is higher than or equal to a preset threshold, the entire text data is considered to be of reliable quality and can proceed directly to the next step of fusion processing; if the score is lower than the threshold, it indicates that there are one or more serious recognition problems in the document, requiring the natural language processing model to correct them.

[0244] By incorporating both character-level recognition reliability and layout-level structural consistency into the scoring system, content that is simply correctly recognized but has obvious positional anomalies, or content with a reasonable layout but significant character recognition errors, can be accurately reflected in the comprehensive score. By assigning different weights to different target elements, the priority of judging key information fields in financial documents can be highlighted, reducing misjudgments caused by relying solely on OCR confidence levels. This ensures that the text data scoring results simultaneously reflect character recognition quality and layout consistency, thereby improving the detection capability of low-confidence text and reducing the probability of erroneous text entering subsequent business processing stages.

[0245] The document processing method based on multimodal feature fusion provided in this application determines the field type corresponding to each target element in the text data based on semantic parsing results, and extracts the actual coordinate region of each target element in the document image from image feature analysis results; obtains the standard coordinate region of each target element based on a preset financial document template, calculates the distance deviation value between the actual coordinate region and the standard coordinate region, and converts it into a layout matching sub-score according to mapping rules, and then summarizes it to obtain the overall layout matching degree; then, the text data is segmented according to target elements, and for each target element, the character confidence score of the character recognition output and the corresponding layout matching sub-score are obtained. The sub-score is then weighted and summed using preset character confidence score weights and layout matching score weights to obtain the comprehensive confidence score of each target element, and finally, the confidence score of the entire text data is obtained by weighted averaging. This solves the problem of misjudgment caused by traditional OCR relying only on character probability and ignoring layout position matching, improving the accuracy of field recognition confidence judgment and reducing recognition errors.

[0246] Figure 4 A schematic diagram of the structure of the document processing apparatus for multimodal feature fusion provided in this application is shown below. Figure 4 As shown, the multimodal feature fusion document processing apparatus 400 provided in this embodiment includes:

[0247] The acquisition module 401 is used to acquire the document image of the financial document to be processed;

[0248] Processing module 402 is used to perform character recognition on document images to obtain text data;

[0249] The processing module 402 is also used to perform image feature analysis on the document image and semantic parsing on the text data;

[0250] The processing module 402 is also used to determine the confidence score of the text data based on the image feature analysis results and the semantic parsing results;

[0251] The processing module 402 is also used to correct text data through a natural language processing model when the confidence score is lower than a preset threshold.

[0252] The processing module 402 is also used to fuse the image feature analysis results with the parsing results obtained by re-semantically parsing the corrected text data to generate structured data.

[0253] Processing module 402 is also used to perform semantic contradiction detection on structured data based on financial knowledge graph, and to determine contradictory fields and their contradiction risk levels;

[0254] The processing module 402 is also used to correct the contradiction field according to the corresponding contradiction risk level, obtain the corrected structured data and output it.

[0255] In one possible implementation, the device further includes: a calculation module 403 and a determination module 404;

[0256] The calculation module 403 is used to calculate the layout matching degree based on the layout structure information in the semantic parsing results and the layout structure information in the image feature analysis results.

[0257] The determination module 404 is used to determine the confidence score of the text data based on the character confidence score and layout matching degree output during character recognition.

[0258] In one possible implementation, the determining module 404 is further configured to determine the fields corresponding to each target element in the text data based on the semantic parsing results;

[0259] The processing module 402 is also used to extract the image coordinate regions of each target element in the document image from the image feature analysis results;

[0260] The determination module 404 is also used to determine the preset coordinate area corresponding to each target element based on the preset financial document template;

[0261] The calculation module 403 is also used to calculate the distance deviation between the image coordinate region of the target element and the preset coordinate region for each target element;

[0262] Processing module 402 is also used to convert the distance deviation value into a layout matching degree sub-score according to a preset mapping rule;

[0263] The calculation module 403 is also used to calculate the layout matching degree based on the layout matching degree sub-score of each target element.

[0264] In one possible implementation, the processing module 402 is further configured to segment the text data according to the target element to obtain at least one target element to be scored.

[0265] The acquisition module 401 is also used to acquire, for each target element, the character confidence score, the corresponding layout matching score, and the preset character confidence score weight and layout matching score weight.

[0266] The calculation module 403 is also used to perform a weighted summation calculation on the character confidence score and the layout matching score based on the character confidence score weight and the layout matching score weight, so as to obtain the comprehensive confidence score of the target element.

[0267] The processing module 402 is also used to perform a weighted average of the comprehensive confidence sub-scores of each target element to obtain the confidence score of the text data.

[0268] In one possible implementation, the processing module 402 is further configured to extract target elements in the text data whose confidence scores are lower than a preset threshold.

[0269] The processing module 402 is also used to locate the semantic anchor of the target element in the document. The semantic anchor includes at least one of the field name of the target element, adjacent target elements, or table headers.

[0270] The processing module 402 is also used to infer the correct value of the target element based on the context information of the semantic anchor and in combination with the financial business logic rules.

[0271] The processing module 402 is also used to replace the incorrect value in the target element with the inferred correct value to obtain the corrected text data.

[0272] In one possible implementation, the processing module 402 is further configured to search for the target entity and entity attributes corresponding to each field in the financial knowledge graph for the fields in the structured data.

[0273] Processing module 402 is also used to perform consistency verification on target entities and entity attributes based on the relationships and constraint rules in the financial knowledge graph;

[0274] The determination module 404 is also used to locate the corresponding contradictory field and contradictory type in the structured data based on the inconsistent entity attributes if the verification result is inconsistent, and to determine the risk level of the contradictory according to the preset risk level mapping rules.

[0275] In one possible implementation, the processing module 402 is further configured to, if the risk level is low risk level, call a preset rule base, match the corresponding correction rule according to the contradiction type, obtain the correction value, and replace the error value in the contradiction field with the correction value to obtain the corrected structured data;

[0276] The processing module 402 is also used to generate an early warning report containing the original value of the contradiction field, the suggested correction value, the contradiction type description and the correction suggestion if the risk level is high risk level, and push the early warning report to the review terminal.

[0277] In one possible implementation, the processing module 402 is further configured to receive a confirmed correction report returned by the audit terminal when the risk level is high risk level.

[0278] The processing module 402 is also used to correct the contradictory fields according to the correction values ​​in the correction report, and to obtain and output the corrected structured data.

[0279] The document processing device based on multimodal feature fusion provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0280] Figure 5 A schematic diagram of the structure of the electronic device provided in this application. Figure 5 As shown, the electronic device 500 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the electronic device 500 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0281] In the specific implementation process, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to execute the above-mentioned document processing method based on multimodal feature fusion.

[0282] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0283] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0284] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0285] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0286] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described document processing method based on multimodal feature fusion.

[0287] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the aforementioned document processing method based on multimodal feature fusion.

[0288] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0289] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0290] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0291] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0292] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0293] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0294] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0295] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A document processing method based on multimodal feature fusion, characterized in that, include: Acquire a document image of the financial document to be processed, and perform character recognition on the document image to obtain text data; Image feature analysis is performed on the document image, and semantic parsing is performed on the text data; Based on the image feature analysis results and semantic parsing results, the confidence score of the text data is determined; When the confidence score is lower than a preset threshold, the text data is corrected using a natural language processing model. The image feature analysis results are fused with the analysis results obtained by re-semantically analyzing the corrected text data to generate structured data; Semantic contradiction detection is performed on the structured data based on financial knowledge graphs to determine contradictory fields and their contradiction risk levels. Based on the corresponding conflict risk level, the conflict field is corrected to obtain the corrected structured data, which is then output.

2. The method according to claim 1, characterized in that, The step of determining the confidence score of the text data based on the image feature analysis results and semantic parsing results includes: Based on the layout structure information in the semantic parsing results and the layout structure information in the image feature analysis results, the layout matching degree is calculated; The confidence score of the text data is determined based on the character confidence score output during character recognition and the layout matching degree.

3. The method according to claim 2, characterized in that, The calculation of layout matching degree based on the layout structure information in the semantic parsing result and the layout structure information in the image feature analysis result includes: Based on the semantic parsing results, determine the fields corresponding to each target element in the text data; Extract the image coordinate regions of each target element in the document image from the image feature analysis results; Based on a preset financial document template, determine the preset coordinate region corresponding to each target element; For each target element, calculate the distance deviation between the image coordinate region of the target element and the preset coordinate region; The distance deviation value is converted into a layout matching score according to a preset mapping rule; The layout matching degree is calculated based on the layout matching degree sub-score of each target element.

4. The method according to claim 2, characterized in that, The determination of the confidence score of the text data based on the character confidence score output during character recognition and the layout matching degree includes: The text data is segmented according to the target element to obtain at least one target element to be scored; For each target element, obtain the character confidence score, the corresponding layout matching score, and the preset character confidence score weight and layout matching score weight. Based on the character confidence weight and the layout matching weight, the character confidence and layout matching sub-scores are weighted and summed to obtain the comprehensive confidence sub-score of the target element. The confidence score of the text data is obtained by weighted averaging of the comprehensive confidence scores of each target element.

5. The method according to claim 1, characterized in that, The step of correcting the text data using a natural language processing model includes: Extract target elements from the text data whose confidence scores are lower than a preset threshold; Locate the semantic anchor point of the target element in the document, wherein the semantic anchor point includes at least one of the field name of the target element, adjacent target elements, or table header; Based on the contextual information of semantic anchors and combined with financial business logic rules, the correct value of the target element is inferred. The incorrect value in the target element is replaced with the inferred correct value to obtain the corrected text data.

6. The method according to claim 1, characterized in that, The financial knowledge graph includes financial business entities, entity attributes, relationships between entities, and constraint rules. The semantic contradiction detection of the structured data based on the financial knowledge graph, determining contradictory fields and their contradiction risk levels, includes: For each field in the structured data, the target entity and entity attributes corresponding to each field are retrieved from the financial knowledge graph; Based on the relationships and constraint rules in the financial knowledge graph, a consistency check is performed on the target entity and its attributes. If the verification result is inconsistent, the corresponding contradictory field and contradiction type in the structured data are located based on the inconsistent entity attributes, and the risk level of the contradiction is determined according to the preset risk level mapping rules.

7. The method according to claim 6, characterized in that, The step of correcting the contradiction field according to the corresponding contradiction risk level to obtain and output the corrected structured data includes: If the risk level is low risk, then the preset rule base is invoked, the corresponding correction rule is matched according to the contradiction type, the correction value is obtained, and the error value in the contradiction field is replaced with the correction value to obtain the corrected structured data; If the risk level is high risk, an early warning report is generated containing the original value of the contradiction field, the suggested correction value, the contradiction type description, and the correction suggestion, and the early warning report is pushed to the review terminal.

8. The method according to claim 7, characterized in that, The method further includes: When the risk level is high risk, receive the confirmed correction report returned by the audit terminal; Based on the correction values ​​in the correction report, the contradictory fields are corrected to obtain corrected structured data, which is then output.

9. A document processing device based on multimodal feature fusion, characterized in that, include: The acquisition module is used to acquire document images of the financial documents to be processed; The processing module is used to perform character recognition on the document image to obtain text data; The processing module is also used to perform image feature analysis on the document image and semantic parsing on the text data; The processing module is also used to determine the confidence score of the text data based on the image feature analysis results and semantic parsing results; The processing module is also used to correct the text data using a natural language processing model when the confidence score is lower than a preset threshold. The processing module is further configured to fuse the image feature analysis results with the parsing results obtained by re-semantically parsing the corrected text data to generate structured data. The processing module is also used to perform semantic contradiction detection on the structured data based on the financial knowledge graph, and to determine the contradictory fields and their contradiction risk levels. The processing module is also used to correct the contradiction field according to the corresponding contradiction risk level, and output the corrected structured data.

10. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-8.

12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-8.

Images