A document recognition method, apparatus, device and medium

Abstract

The application relates to a document recognition method, device, equipment and medium. It relates to the technical field of computers, and the method comprises the following steps: target information is extracted from multi-source heterogeneous document data; wherein the target information comprises at least one of text information, image features, position information and time sequence information; the target information is subjected to alignment processing based on preset rule constraints to obtain an alignment deviation result corresponding to the target information; multi-modal features corresponding to the target information are determined based on the alignment deviation result; and the document data is recognized based on the alignment deviation result and the multi-modal features.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and specifically to a document recognition method, apparatus, device, and medium. Background Technology

[0002] With the widespread adoption of digital office tools, document types are becoming increasingly diverse and heterogeneous, with common formats including text documents and documents combining text and images. In document processing scenarios, it is often necessary to identify document content to meet business approval requirements.

[0003] In related technologies, for document content recognition, semantic alignment of the document content is performed to obtain alignment deviation results, and document recognition is performed based on the alignment deviation results.

[0004] However, relying solely on semantic alignment operations to obtain alignment deviation results leads to inaccurate alignment deviation results, thus failing to accurately identify documents.

[0005] Therefore, there is an urgent need for a method that can accurately identify documents. Summary of the Invention

[0006] The technical problem to be solved by the present invention is that the alignment deviation result obtained by relying solely on semantic alignment operation is inaccurate, thus making it impossible to accurately identify documents.

[0007] The technical solution of the present invention to solve the above-mentioned technical problems is as follows: a document recognition method, the method comprising: extracting target information from multi-source heterogeneous document data; wherein the target information includes at least one of text information, image features, location information and temporal information; performing alignment processing on the target information based on preset rule constraints to obtain an alignment deviation result corresponding to the target information; determining the multimodal features corresponding to the target information based on the alignment deviation result; and recognizing the document data based on the alignment deviation result and the multimodal features.

[0008] The beneficial effects of this invention are as follows: By first extracting target information containing text information, image features, location information, and temporal information from multi-source heterogeneous document data, and then aligning the extracted target information based on preset rule constraints, the matching degree of multiple aspects such as text, image, location, and temporal information can be comprehensively considered, thereby obtaining a more comprehensive and accurate alignment deviation result. This solves the problem of inaccurate alignment deviation results obtained solely through semantic alignment in related technologies. Based on the accurate alignment deviation result, multimodal features are determined, and then the document data is identified by combining the alignment deviation result and multimodal features. The accuracy of document recognition is improved through multi-dimensional verification.

[0009] Based on the above technical solution, the present invention can be further improved as follows.

[0010] Furthermore, target information is extracted from multi-source heterogeneous document data, including: extracting and processing the multi-source heterogeneous document data to obtain text confidence and information to be processed; wherein, the information to be processed includes: text information to be processed, image information to be processed, location information to be processed, and temporal information to be processed; based on the text confidence, anomaly detection processing is performed on the information to be processed to obtain target information; wherein, the anomaly detection processing includes: when the text recognition confidence is lower than a preset confidence threshold or the text information to be processed is invalid, the image information to be processed is identified as target information; when the preset rule constraint is abnormal, the information to be processed is identified as target information.

[0011] The beneficial effect of adopting the above-mentioned further scheme is that by extracting and processing multi-source heterogeneous document data, the specific types of information to be processed can be clearly distinguished, making the text information, image information, location information, and time sequence information to be processed clearly classified. The validity of the information to be processed can be judged by the text confidence score, avoiding interference from invalid information, thereby improving the accuracy of document recognition.

[0012] Furthermore, the method also includes: converting the business rules to be processed into computable constraints, configuring judgment thresholds and weights for the computable constraints to form a set of rule constraints; and determining preset rule constraints based on predefined rule templates, historical document data, and the set of rule constraints.

[0013] The beneficial effect of adopting the above-mentioned further solution is that after converting the business rules to be processed into computable and executable constraints, a corresponding judgment threshold and weight parameter are configured for each constraint. The judgment threshold is used to define the execution standard of the constraint, and the weight is used to distinguish the priority of each constraint. Together, they form a set of rule constraints, which ensures the rationality and adaptability of the rule constraints, thereby improving the document recognition accuracy.

[0014] Furthermore, based on predefined rule templates, historical document data, and a set of rule constraints, preset rule constraints are determined, including: optimizing the judgment thresholds and weights of each rule constraint in the set of rule constraints based on the predefined rule templates and historical document data to obtain a target set of rule constraints; detecting whether there are logically conflicting rule constraints in the target set of rule constraints; if there are logically conflicting rule constraints in the target set of rule constraints, prioritizing them according to their weights to obtain a ranking result; determining the rule satisfaction of the logically conflicting rule constraints based on the ranking result; determining the rule constraints used for alignment processing from the logically conflicting rule constraints based on the rule satisfaction; and determining the preset rule constraints based on the rule constraints used for alignment processing and the rule constraints without logical conflicts.

[0015] The beneficial effect of adopting the above-mentioned further solution is that by combining predefined rule templates and historical document data, the judgment thresholds and weights in the rule constraint set can be optimized in a targeted manner. This makes each rule constraint more in line with the actual document recognition scenario, improves the rationality and executability of the rules, and thus obtains a more suitable set of target rule constraints. By detecting logical conflicts in the target rule constraint set and prioritizing them according to weights, contradictions between rules can be accurately identified and resolved efficiently. Then, the rule satisfaction is determined based on the ranking results, and effective rules for alignment processing are selected. Finally, the conflict-free rules and the selected conflicting rules are integrated to ensure that the final determined preset rule constraints have no logical contradictions, thereby improving the accuracy of document recognition.

[0016] Furthermore, based on preset rule constraints, alignment processing is performed on the target information to obtain the alignment deviation result corresponding to the target information, including: vector encoding of text information based on preset rule constraints to obtain a first vector code; generating text descriptions for image regions corresponding to image features and vector encoding the text descriptions to obtain a second vector code; determining the matching similarity based on the first and second vector codes; determining the semantic alignment deviation based on the matching similarity; performing spatial matching calibration based on the text position information in the target information and the image region position to obtain the first bounding box coordinates of the text information and the second bounding box coordinates of the image region; determining the intersection-union ratio (IU / I) based on the first and second bounding box coordinates; determining the spatial alignment deviation based on the IU / I and IU threshold; determining the time difference between the document generation time and upload time corresponding to the target information; determining the temporal alignment deviation based on the time difference and the time difference threshold; and forming the alignment deviation result corresponding to the target information based on the semantic alignment deviation, spatial alignment deviation, and temporal alignment deviation.

[0017] The beneficial effects of the above-mentioned further scheme are as follows: by vector encoding the text information to obtain the first vector code, and generating text descriptions of the image regions corresponding to image features and then vector encoding them to obtain the second vector code, the matching similarity is determined based on the two types of vector codes, thereby accurately quantifying the semantic alignment deviation and ensuring semantic consistency. Furthermore, by obtaining the coordinates of the first bounding box corresponding to the text information and the coordinates of the second bounding box corresponding to the image region, the intersection-union ratio (IU) of the two is calculated, and the spatial alignment deviation is determined by combining the IU threshold, ensuring spatial matching accuracy. Moreover, by determining the time difference between the document generation time and the upload time, and combining it with a time difference threshold, the temporal alignment deviation is quantified. Finally, the three types of deviations are integrated to form a complete alignment deviation result. Then, based on the semantic alignment deviation, spatial alignment deviation, and temporal alignment deviation, the document recognition accuracy can be improved.

[0018] Furthermore, based on the alignment deviation results, the multimodal features corresponding to the target information are determined, including: converting the alignment deviation results into constraint information for feature fusion; adjusting the attention weights based on the constraint information to obtain the target attention weights; and performing feature fusion on the target information based on the target attention weights to obtain the multimodal features corresponding to the target information.

[0019] The beneficial effect of adopting the above-mentioned further scheme is that it transforms the alignment deviation result into the constraint information required for feature fusion, ensuring the compatibility between the constraint information and the target information; by adjusting the attention weight based on the constraint information, it can selectively highlight the effective target information that conforms to the preset rule constraints and weaken abnormal or invalid information, avoiding irrelevant information from interfering with the feature fusion process; and then, based on the adjusted target attention weight, it can perform feature fusion on the target information, which can achieve effective complementarity of multiple types of target information such as text, images, and time series, thereby improving the document recognition accuracy.

[0020] Furthermore, based on the alignment deviation results and multimodal features, document data is identified, including: determining the evidence strength information of the document data based on the multimodal features; determining the risk information of the document data based on the alignment deviation results; and identifying the document data based on the evidence strength information and the risk information.

[0021] The beneficial effect of adopting the above-mentioned further scheme is that it determines the evidence strength information of document data based on multimodal features, and then determines the risk information of document data based on the alignment deviation results. It accurately captures the potential deviations in the semantic, spatial and temporal three-dimensional alignment process, effectively avoids the deviations caused by single-based identification, and thus improves the accuracy of document recognition.

[0022] Furthermore, the present invention provides a document recognition device, comprising: a first processing module for extracting target information from multi-source heterogeneous document data; wherein the target information includes at least one of text information, image features, location information, and temporal information; a second processing module for aligning the target information based on preset rule constraints to obtain an alignment deviation result corresponding to the target information; a determination module for determining multimodal features corresponding to the target information based on the alignment deviation result; and a recognition module for recognizing the document data based on the alignment deviation result and the multimodal features.

[0023] Furthermore, this application provides a computer device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the document recognition method of the first aspect or any corresponding embodiment described above.

[0024] Furthermore, this application provides a computer-readable storage medium storing computer instructions for causing a computer to execute the document recognition method of the first aspect or any corresponding embodiment described above.

[0025] Furthermore, this application provides a computer program product, including computer instructions for causing a computer to execute the document recognition method of the first aspect described above or any corresponding embodiment thereof. Attached Figure Description

[0026] Figure 1 A flowchart illustrating the document recognition method provided by this invention; Figure 2 A flowchart illustrating another document recognition method provided by the present invention; Figure 3 A flowchart illustrating another document recognition method provided by the present invention; Figure 4 This is a schematic diagram of the cross-modal attention fusion network provided by the present invention; Figure 5 A flowchart illustrating another document recognition method provided by the present invention; Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0027] The principles and features of the present invention are described below. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0028] With the widespread adoption of digital office tools, document types are becoming increasingly diverse and heterogeneous, with common formats including text documents and documents combining text and images. In document processing scenarios, it is often necessary to identify document content to meet business approval requirements.

[0029] In related technologies, for document content recognition, semantic alignment of the document content is performed to obtain alignment deviation results, and document recognition is performed based on the alignment deviation results.

[0030] Especially when it comes to debt collection, it is necessary to rely on evidence from multiple sources, such as scanned copies of contracts, bank statements, SMS screenshots, and handwritten explanations of the outstanding debt, to decide on subsequent collection measures.

[0031] Debt collectors are not concerned with simply reading the text aloud, but with determining whether the evidence is complete, authentic, credible, and consistent with the information, ultimately generating an evidence credibility profile that can be used automatically by the system.

[0032] However, relying solely on semantic alignment operations results in inaccurate alignment deviations, which in turn fail to accurately identify documents and thus prevent the generation of a credibility profile of evidence that can be used automatically by the system.

[0033] Based on this, the present invention provides a document recognition method. By first extracting target information containing text information, image features, location information, and temporal information from multi-source heterogeneous document data, and then aligning the extracted target information based on preset rule constraints, the method can comprehensively consider the matching degree of multiple aspects of information such as text, image, location, and temporal sequence, thereby obtaining a more comprehensive and accurate alignment deviation result. This solves the problem of inaccurate alignment deviation results obtained solely through semantic alignment in related technologies. Based on the accurate alignment deviation result, multimodal features are determined, and then the document data is recognized by combining the alignment deviation result and multimodal features. The document recognition accuracy is improved through multi-dimensional verification.

[0034] According to an embodiment of the present invention, a document recognition method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0035] This embodiment provides a document recognition method that can be used in computer devices such as computers and servers. Figure 1 This is a flowchart of a document recognition method according to an embodiment of this application, such as... Figure 1 As shown, the process includes the following steps: Step S101: Extract target information from multi-source heterogeneous document data; wherein the target information includes at least one of text information, image features, location information and temporal information.

[0036] Multi-source heterogeneous document data can be understood as a collection of various documents from different sources and in different formats. The sources can include uploads from business systems, manual input, imports from third-party interfaces, etc., and the formats can cover PDF, images (JPG, PNG, etc.), Word, Excel, etc., and the documents can contain multiple types of information such as text, images, location, and time series.

[0037] Target information can be understood as information extracted from multi-source heterogeneous document data for subsequent alignment processing, feature fusion, and document recognition. Target information includes at least one of the following: textual information (such as the text content in the document, including body text, annotations, signatures, etc.), image features (such as the core features of visual information such as images, pictures, seals, and signatures in the document), location information (such as the specific coordinates of text and images in the document, bounding box positions, etc.), and temporal information (such as time-related data such as the document's generation time, upload time, and processing time).

[0038] Specifically, preprocessing and extraction operations are performed on multi-source heterogeneous document data to obtain target information.

[0039] As an example, preprocessing operations may include the following steps: The document data is processed sequentially as follows: format validation, image denoising, skew correction, page segmentation, text recognition, text position extraction, and extraction of timestamps and uploader information.

[0040] Then, the text content, image blocks, location coordinates and temporal information obtained from each preprocessing stage are integrated to form structured target information.

[0041] Step S102: Based on preset rule constraints, the target information is aligned to obtain the alignment deviation result corresponding to the target information.

[0042] Preset rule constraints can be understood as converting specific business rules into calculable and executable constraints based on business needs, which are used to standardize target information alignment processing, feature fusion, and document recognition.

[0043] Alignment deviation results can be understood as deviation data of the degree of matching of target information in various dimensions during the alignment process. These can include semantic alignment deviation, spatial alignment deviation, temporal alignment deviation, etc., without specific limitations here.

[0044] Specifically, based on preset rule constraints, the extracted target information is aligned and verified. Deviation data for each dimension is obtained through quantitative calculation, and then integrated to form a complete alignment deviation result.

[0045] As an example, the target information extracted in step S101 is subjected to alignment verification in three dimensions: semantic, spatial, and temporal. Semantic matching calibration can be understood as performing semantic matching between text information and image features in the target information, calculating the degree of matching, and quantifying the semantic alignment deviation. Spatial matching calibration can be understood as performing spatial matching between text location information and image region location in the target information, calculating the degree of positional overlap, and quantifying the spatial alignment deviation. Temporal relationship verification can be understood as verifying the rationality of temporal information in the target information, calculating relevant time intervals, and quantifying the temporal alignment deviation. Deviation integration can be understood as integrating the alignment deviations in the semantic, spatial, and temporal dimensions to form a complete alignment deviation result for subsequent multimodal feature determination.

[0046] Step S103: Based on the alignment deviation results, determine the multimodal features corresponding to the target information.

[0047] Multimodal features can be understood as features obtained by fusing target information based on alignment deviation results.

[0048] Specifically, based on the alignment deviation results, the feature fusion process is optimized to integrate various single-modal features in the target information and form a unified multimodal feature.

[0049] As an example, the alignment deviation results are converted into constraint information that can be directly used in feature fusion. During the cross-modal attention calculation process, the attention weight allocation strategy is dynamically adjusted based on the constraint information. Target information that meets the preset rule constraints is given higher weight. Based on the adjusted attention weights, the text information, image features, and temporal information in the target information are jointly fused. Through feature alignment, dimensional unification, and information complementarity, a unified multimodal feature is formed.

[0050] Step S104: Based on the alignment deviation results and multimodal features, the document data is identified.

[0051] After determining the alignment deviation and multimodal features, the document data can be identified.

[0052] As an example, the evidence strength of document data is calculated based on multimodal features, the risk information of document data is calculated based on alignment deviation results, and the credibility assessment result is generated by combining the evidence strength, risk information, contribution weight of each modality and audit mark. The document data is then identified based on the credibility assessment result.

[0053] The document recognition method provided by this invention first extracts target information containing text information, image features, location information, and temporal information from multi-source heterogeneous document data. Then, based on preset rule constraints, it performs alignment processing on the extracted target information. This method can comprehensively consider the matching degree of multiple aspects of information such as text, image, location, and temporal sequence, thereby obtaining a more comprehensive and accurate alignment deviation result. This solves the problem of inaccurate alignment deviation results obtained solely through semantic alignment in related technologies. Based on the accurate alignment deviation result, multimodal features are determined. Then, the alignment deviation result and multimodal features are combined to recognize document data. The document recognition accuracy is improved through multi-dimensional verification.

[0054] Based on step S101 of this embodiment, extracting target information from multi-source heterogeneous document data, such as... Figure 2 Implementation of steps S2011 to S2012: Step S2011: Extract and process the multi-source heterogeneous document data to obtain text confidence and information to be processed; wherein, the information to be processed includes: text information to be processed, image information to be processed, location information to be processed, and time sequence information to be processed.

[0055] Text confidence can be understood as the reliability of text recognition results, used to determine whether the text information to be processed is valid.

[0056] The text information to be processed can be understood as textual information in the information to be processed, including the main text, annotations, descriptions and other textual content in various documents.

[0057] The image information to be processed can be understood as visual information in the information to be processed, including visual content such as images, signatures, and photos in the document.

[0058] The location information to be processed can be understood as location-related data in the information to be processed, such as the specific coordinates of text and images in the document, the position of the bounding box, etc.

[0059] The pending time sequence information can be understood as time-related data in the pending information, including document generation time, upload time, etc.

[0060] Specifically, extraction and processing operations are performed on multi-source heterogeneous document data. The core output of this operation consists of two types of results: text confidence and unprocessed information. The unprocessed information is specifically composed of unprocessed text information, unprocessed image information, unprocessed location information, and unprocessed temporal information.

[0061] As an example, this paper extracts and processes multi-source heterogeneous document data (PDF documents and JPG image documents). For PDF documents, it extracts text information (text content within the document), image information (signature images within the document), location information (boundary coordinates of text and signatures), and time sequence information (document generation time and upload time). For JPG image documents, it extracts image information (image content), location information (coordinates of key regions within the image), and time sequence information (image capture time and upload time). Simultaneously, it calculates the text confidence scores of the text information in both types of documents, ultimately obtaining the text confidence scores and a set of unprocessed information containing the four types of unprocessed information.

[0062] As an example, this paper extracts and processes heterogeneous document data from multiple sources (Word documents and Excel documents). For Word documents, it extracts the text information to be processed (document body and annotations), the location information to be processed (text line coordinates), and the time sequence information to be processed (document generation and modification time). For Excel documents, it extracts the text information to be processed (data text in the table), the location information to be processed (table cell coordinates), and the time sequence information to be processed (table generation and upload time). At the same time, it calculates the text confidence of the text information to be processed in both types of documents, and finally obtains the text confidence and the set of unprocessed information containing the four types of unprocessed information.

[0063] Step S2012: Based on text confidence, perform anomaly detection processing on the information to be processed to obtain target information; wherein, the anomaly detection processing includes: when the text recognition confidence is lower than a preset confidence threshold or the text information to be processed is invalid, the image information to be processed is determined as target information; when the preset rule constraint is abnormal, the information to be processed is determined as target information.

[0064] The preset confidence threshold can be understood as a pre-set confidence threshold. Specifically, the preset confidence threshold is used to determine whether the text confidence level meets the standard and whether the text information to be processed is valid.

[0065] Specifically, based on text confidence levels, anomaly detection is performed on the information to be processed, and the target information is finally output. This can include two types of scenarios.

[0066] The first type of abnormal scenario: When the text recognition confidence is lower than the preset confidence threshold, or the text information to be processed is invalid, the processing operation is to determine the image information to be processed as the target information; The second type of abnormal scenario: When the preset rule constraint is abnormal, the processing operation is to determine the information to be processed as the target information; As an example, in step S2011, the text confidence score is 0.6, and the preset confidence threshold is 0.7. The text confidence score is lower than the preset confidence threshold, which belongs to the first type of scenario for anomaly detection processing. Therefore, the image information to be processed obtained in step S2011 is determined as the target information.

[0067] As an example, if the text information to be processed obtained in step S2011 contains unrecognizable invalid content (the text information to be processed is invalid), it belongs to the first type of anomaly detection processing scenario. According to the weighting rule, the image information to be processed obtained in step S2011 is determined as the target information. In another case, if the text confidence level obtained in step S2011 is 0.8 (higher than the preset confidence threshold) and the text information to be processed is valid, but the preset rule constraints cannot be loaded normally (the preset rule constraints are abnormal), it belongs to the second type of anomaly detection processing scenario. According to the weighting rule, the information to be processed obtained in step S2011 is determined as the target information.

[0068] In one possible implementation scenario, multi-source heterogeneous document data (PDF collection notices and JPG ID card photos) are extracted and processed. The PDF collection notices extract unprocessed text information (borrower's name, collection amount), unprocessed image information (collection signature), unprocessed location information (text and signature coordinates), and unprocessed time sequence information (generation and upload time). Similarly, the JPG ID card photos extract unprocessed image information (headshot and text area), unprocessed location information (headshot and text area coordinates), and unprocessed time sequence information (shooting and upload time). The text confidence score for the unprocessed text information in the PDF collection notices is calculated to be 0.8, and the text confidence score for the unprocessed text information in the ID card photos is calculated to be 0.6. Finally, the text confidence scores (0.8 and 0.6) and a set of unprocessed information containing four types of unprocessed information are obtained.

[0069] Anomaly detection processing is carried out based on the above text confidence scores, with a preset confidence threshold of 0.7. Among them, the text confidence score of the ID card photo is 0.6, which is lower than the preset confidence threshold, belonging to the first type of anomaly scenario, and the image information to be processed is determined as the target information. The text confidence score of the PDF collection notice is 0.8, which is higher than the preset confidence threshold, the text information to be processed is valid, and the preset rule constraints are normal, with no anomalies, so the corresponding information to be processed is determined as the target information. Finally, the target information is obtained.

[0070] In one possible implementation, combining Figure 3 As shown, this embodiment of the invention provides yet another document recognition method. This method may include: The data is processed according to the following steps: format validation, image denoising, page segmentation, OCR recognition, timestamp, and uploader record. The final output includes text lines and locations, image blocks and coordinates. This structured data can be directly used by the rules and alignment modules, and all evidence is uniformly converted into a combination of text, image, and time to ensure data usability. Error handling and degradation strategies, as well as anomaly detection, are also implemented. The specific process is as follows: OCR recognition failure handling: If the detection condition is OCR confidence < 0.5 or text extraction is empty, the image feature-dominated mode is triggered, and only the image visual features are used for subsequent processing. The OCR_FAILED mark is recorded in the audit trajectory, and the text confidence is reduced to 0.3. Image corruption or format abnormality handling: The detection condition is image decoding failure or image size <100×100 pixels, return error code ERR_IMAGE_INVALID, trigger the replacement policy, and record the IMAGE_INVALID mark in the audit track; Handling rule base loading failure: The detection condition is that the rule base file does not exist or the rule format is parsed incorrectly. The default rule set (containing 3 basic rules: signature exists, amount is not negative, time is reasonable) is used, and the RULE_FALLBACK flag is recorded in the audit track. Anomaly handling in fusion network inference: The detection condition is NaN in the model output or mismatch in feature vector dimensions. The rule engine is used to judge the pattern separately, skip the fusion network, and directly output the rule-based recognition result. The FUSION_FAILED mark is recorded in the audit trajectory, and the confidence level is adjusted to [0.5±0.2]. Based on the above anomaly detection scenario, the first type of anomaly processing is performed on the ID card photo (text confidence 0.6 < 0.7), and the image information to be processed is determined as the target information; the PDF collection notice (text confidence 0.8 ≥ 0.7) is determined as the target information because there are no anomalies; the target information is finally obtained and used for subsequent alignment processing.

[0071] The document recognition method provided in this invention extracts and processes multi-source heterogeneous document data, clearly distinguishes the specific types of information to be processed, and makes the text information, image information, location information, and time sequence information to be processed clearly classified. The validity of the information to be processed is judged by the text confidence score, avoiding interference from invalid information, thereby improving the accuracy of document recognition.

[0072] In order to improve the accuracy of the alignment deviation results, in one possible implementation, the above document recognition method may also include steps a1 to a2.

[0073] Step a1: Convert the business rules to be processed into computable constraints, and configure judgment thresholds and weights for the computable constraints to form a set of rule constraints.

[0074] Computable constraints can be understood as quantifiable, executable, and usable conditions for rule constraint processing, formed after transforming the business rules to be processed.

[0075] A threshold can be understood as a criterion for determining a condition based on calculable constraints. Examples include image size thresholds and OCR confidence thresholds.

[0076] Weights can be understood as priority parameters for calculable constraint settings, used to distinguish the importance of different constraints.

[0077] The rule constraint set can be understood as a set formed by integrating all computable constraints with their corresponding judgment thresholds and weights.

[0078] Specifically, the business rules to be processed (such as the existence of signatures, non-negative amounts, and reasonable timing) are converted into quantifiable, executable, and computable constraints. For each computable constraint, a corresponding judgment threshold and weight are configured. The judgment threshold is used to define the execution standard of the constraint (such as the threshold setting logic for OCR confidence < 0.5), and the weight is used to distinguish the priority of each constraint (such as the weight of core business rules being higher than that of ordinary rules). All computable constraints with configured judgment thresholds and weights are integrated to form a set of rule constraints.

[0079] Step a2: Determine the preset rule constraints based on the predefined rule template, historical document data, and rule constraint set.

[0080] Historical document data can be understood as multi-source heterogeneous document data that has been processed in the past, rule execution records, exception handling records, etc., such as OCR recognition records, rule loading records, audit trails and other data of past documents.

[0081] Specifically, based on predefined rule templates and historical document data, the set of rule constraints formed in step a1 is optimized to ultimately determine the preset rule constraints. The specific processing procedure is described in detail below.

[0082] Specifically, step a2 above may include: Step a21: Based on the predefined rule template and historical document data, optimize the judgment threshold and weight of each rule constraint in the rule constraint set to obtain the target rule constraint set.

[0083] Predefined rule templates can be understood as templates that are pre-set with standardized constraint formats, weight allocation logic, and judgment threshold ranges.

[0084] The target rule constraint set can be understood as a temporary rule constraint set formed after optimizing the judgment thresholds and weights of each rule constraint in the rule constraint set.

[0085] Specifically, using predefined rule templates as a standard and referring to historical document data, the judgment threshold and weight of each rule constraint in the rule constraint set are verified one by one, and unreasonable parameters are adjusted and optimized: combining the success or failure records of rule execution in historical documents and abnormal handling scenarios (such as image corruption, OCR recognition failure); adjusting the weight of each rule constraint according to historical business priorities, and increasing the priority of rule constraints related to core business; after optimization, integrating all adjusted rule constraints to form the target rule constraint set.

[0086] In a feasible scenario, the predefined rule template has a business rule constraint weight ≥ 0.3, an OCR confidence threshold range of 0.6-0.8, and historical document data containing rule execution records of 100 past collection documents (15 of which failed due to an OCR confidence of 0.65, and 10 failed due to the signature area being slightly smaller than 50×50 pixels). The rule constraint set contains 3 rule constraints: OCR confidence ≥ 0.7 (threshold 0.7, weight 0.3); signature area ≥ 50×50 pixels (threshold 50×50 pixels, weight 0.3); collection amount ≥ 0 (threshold 0, weight 0.4). Based on the template and historical data optimization: the OCR confidence threshold is adjusted to 0.65, the signature area threshold is adjusted to 45×45 pixels, and the weights remain unchanged. After optimization, the sets are integrated to form the target rule constraint set.

[0087] Step a22: Detect whether there are logically conflicting rule constraints in the target rule constraint set.

[0088] Logically conflicting rule constraints can be understood as situations in the target rule constraint set where the judgment logic of two or more rule constraints contradicts each other and cannot be satisfied simultaneously (such as two rules having opposite threshold requirements for the same indicator).

[0089] Specifically, each rule constraint in the target rule constraint set is checked one by one to detect whether there are any logical conflicts.

[0090] Step a23: If there are logically conflicting rule constraints in the target rule constraint set, sort them according to their weights to obtain the sorting result.

[0091] When logically conflicting rule constraints are detected in the target rule constraint set, all conflicting rule constraints are extracted, and the weight value corresponding to each rule constraint is obtained. These conflicting rule constraints are sorted in descending order of weight value, with higher weights indicating higher priority. After sorting, a sorting result is generated, the priority order of each conflicting rule is determined, and the sorting basis is recorded.

[0092] In one possible implementation scenario, two rule constraints with logical conflicts are detected: (1) OCR confidence ≥ 0.65 (weight 0.4); (2) OCR confidence < 0.6 (weight 0.3); sorted from high to low weight, the sorting result is: the weight of (1) > the weight of (2), indicating that the priority of (1) is higher.

[0093] Step a24: Based on the sorting results, determine the rule satisfaction of rule constraints with logical conflicts.

[0094] Rule satisfaction can be understood as a quantitative indicator (range 0-1) that reflects the adaptability of conflicting rules, calculated based on the ranking results (priority) and the success rate of rule execution in historical document data.

[0095] Specifically, by combining the execution success rate of each conflicting rule constraint in historical document data, the rule satisfaction of each logically conflicting rule constraint is calculated: the higher the priority of the rule constraint, the higher the basic satisfaction weight; at the same time, combined with the historical execution success rate (e.g., the higher the execution success rate of a rule in past document processing, the higher the satisfaction), the rule satisfaction of each conflicting rule is comprehensively calculated. The quantitative index takes a value of 0-1. The higher the value, the stronger the rule adaptability and the more it meets business needs.

[0096] Step a25: Based on rule satisfaction, determine the rule constraints used for alignment processing from the rule constraints with logical conflicts.

[0097] The satisfaction level of logically conflicting rule constraints is compared with the threshold; rule constraints with a satisfaction level higher than the threshold are selected. If there are multiple conflicting rules with a satisfaction level higher than the threshold, the rule constraint with higher priority (ranked higher in the sorting results) is selected first; if the satisfaction level of all conflicting rules is lower than the threshold, the default rule is used as a substitute according to the rule degradation strategy; finally, one or more rule constraints without logical conflicts and adapted to business needs are determined.

[0098] Step a26: Determine the preset rule constraints based on the rule constraints used for alignment processing and the rule constraints that do not have logical conflicts.

[0099] The rule constraints used for alignment processing (the filtering result of step a25) and the rule constraints without logical conflicts (the detection result of step a22) are integrated, and the integration of all rule constraints is checked to see if there are any new logical conflicts. If there are new conflicts, the logic of steps a23-a25 is referred to again for processing. If there are no new conflicts, all rule constraints are integrated to form the preset rule constraints.

[0100] In one possible implementation, for steps a21 to a26 above, the present invention provides a document recognition method, which includes the following steps: The rule base maintained by the business department, such as contracts needing an official seal and it being located in the upper right corner of the signature, the difference between amounts across pages being ≤1 yuan, and SMS screenshot timestamps falling within 10 minutes of the call, serves as the base rules for conversion and is used to form calculable constraints.

[0101] The original business rules in the rule base are written into quantifiable and executable functions to achieve formal transformation of the rules. For example, the signature location rule is transformed into rule_stamp=IoU(detected signature, expected area)≥0.75. This transformation process converts the original business rules into computable constraints, and at the same time, a corresponding judgment threshold and weight are configured for each transformed rule. The threshold, weight, and current satisfaction of each rule form the rule constraint set (R_rule).

[0102] The rule formalization process needs to combine historical document data (such as compliance samples and business records) to determine thresholds and configure weights, while reserving an adjustable range for thresholds to adapt to subsequent optimization needs. The specific implementation is as follows: (1) Signature position rules (rule_stamp): Expected area determination: Based on the contract template library, extract the standard signature position (located in the lower right corner of the contract, with coordinates [width×0.7, height×0.8, width×0.95, height×0.95]). Dynamic adjustment: If a signature is detected, calculate the IoU between the detection box and the expected area to provide data support for subsequent threshold optimization; Computable function: rule_stamp=IoU(detected_seal_bbox,expected_region)≥0.75 (computable constraint); Weighting configuration: w1=0.3 (the location of the signature has a significant impact on the validity of the evidence; the weight is used to distinguish the priority of rules). Threshold selection basis: By statistically analyzing 1000 compliant contract samples (historical document data), it is found that 95% of the signature position IoU is ≥0.75, so 0.75 is selected as the initial threshold; this threshold can be fine-tuned according to different contract types (such as A4 contracts and A3 contracts) in combination with historical data, with an adjustment range of [0.70, 0.80] to meet adaptation and optimization requirements.

[0103] (2) Amount consistency rule (rule_amount): Amount extraction method: Identify the amount pattern in the text through OCR (regular expression: \d+\.?\d* yuan), and determine the position of the contract amount field in combination with layout analysis; Cross-page consistency: For multi-page documents, extract the amount field of each page and calculate the sum; Computable function: rule_amount=abs(sum(amounts_per_page)-contract_total_amount)≤1.0 (computable constraint condition); Weight configuration: w2=0.25 (used to distinguish rule priorities and provide a basis for subsequent conflict resolution); Threshold selection basis: Considering OCR recognition errors and rounding, a difference range of 1 yuan is tolerable; this threshold is determined by comparing the OCR recognition results of 1000 contract samples with the manually labeled results (historical document data), achieving an accuracy of 98.5%, and can be further optimized in combination with subsequent historical data.

[0104] (3) Time chain rule (rule_time): Timestamp extraction: Extract the upload time from metadata, and extract the receipt time from the OCR text (through time pattern matching: \d{4}-\d{2}-\d{2}\d{2}:\d{2}:\d{2}); Time difference calculation: Δt=|upload_time-receipt_time| (unit: minute); Computable function: rule_time=Δt≤5.0 or sequence_diff≤2 (computable constraint condition); Weight configuration: w3=0.2 (used to distinguish rule priorities and provide a basis for subsequent conflict resolution); Threshold selection basis: Based on business scenario analysis, in the normal business process, the difference between the receipt upload time and the receipt generation time is usually within 5 minutes; this threshold is determined by analyzing 10000 normal business records (historical document data), achieving a coverage rate of 99.2%, and can be dynamically optimized in combination with subsequent historical data.

[0105] When multiple rules in the rule constraint set have logical conflicts, the following processing mechanism is adopted to realize conflict detection, sorting, satisfaction calculation and conflict resolution, and finally select the effective rules: Priority mechanism: Rules are sorted by weight, with rules having higher weights having higher priority. The sorting result serves as the basis for subsequent satisfaction calculations. Conflict detection: Logically verify all rules in the rule constraint set. When rule A is satisfied but rule B is not satisfied, and there is a logical conflict between the two, a conflict flag is triggered to clarify the scope of the conflicting rules. Conflict resolution: A weighted voting mechanism is adopted, which combines rule satisfaction and weight to calculate rule satisfaction. The calculation formula is rule satisfaction = Σ(rule satisfaction × rule weight) / Σ(rule weight). Rule satisfaction serves as the core criterion for selecting effective rules. Rule filtering: A preset rule satisfaction threshold is set, and rules with a satisfaction level higher than the threshold are selected as valid rules; if there are multiple conflicting rules with high satisfaction levels, the rule with higher priority (higher weight) is selected first; if the satisfaction level of all conflicting rules is lower than the threshold, the default rule can be used as a substitute. Dynamic updates: The rule base supports online updates. Newly added rules are automatically checked for conflicts. Once a conflict is detected, the rules are sorted, satisfaction calculated, and filtered according to the mechanism described above.

[0106] The document recognition method provided by this invention optimizes the judgment thresholds and weights in the rule constraint set by combining predefined rule templates and historical document data. This makes each rule constraint more suitable for the actual document recognition scenario, improves the rationality and executability of the rules, and thus obtains a more suitable set of target rule constraints. By detecting logical conflicts in the target rule constraint set and prioritizing them by weight, the method can accurately identify and efficiently resolve contradictions between rules. Based on the ranking results, the method determines the rule satisfaction level and then selects effective rules for alignment processing. Finally, it integrates the conflict-free rules with the selected conflicting rules to ensure that the final determined preset rule constraints are logically consistent, thereby improving the accuracy of document recognition.

[0107] In step S102 of this embodiment, the alignment of the target information based on the preset rule constraints to obtain the alignment deviation result corresponding to the target information can be achieved by steps S4021 to S4030.

[0108] Step S4021: Based on preset rule constraints, the text information is vector-encoded to obtain the first vector code.

[0109] The first vector encoding can be understood as the vector data obtained by encoding text information using vector encoding.

[0110] Specifically, the preset rule constraints, after optimization and conflict handling, are retrieved to clarify the encoding range and format requirements of the text information; the text information in the target information is preprocessed, and according to the requirements of the preset rule constraints, the text information is converted into a fixed-dimensional vector using a vector encoding algorithm to finally obtain the first vector code.

[0111] Step S4022: Generate text descriptions for image regions corresponding to image features, and perform vector encoding on the text descriptions to obtain a second vector code.

[0112] Text description can be understood as natural text that accurately describes the content of an image, generated based on image features and visual information of image regions. Second vector encoding can be understood as vector data obtained by encoding the text description of the image region.

[0113] Specifically, image features and corresponding image regions are extracted from the target information. Based on the visual content of the image regions (such as objects, text, and layout within the regions), an image description generation algorithm is used to generate text descriptions that accurately reflect the image features. Referring to the vector encoding standard in step S4021, the generated text descriptions are vector encoded to obtain a second vector encoding with the same dimension as the first vector encoding.

[0114] Step S4023: Determine the matching similarity based on the first vector encoding and the second vector encoding.

[0115] Matching similarity can be understood as the degree of similarity between the first vector encoding and the second vector encoding.

[0116] Specifically, a vector similarity calculation algorithm can be used to calculate the similarity between two vectors, eliminating the error caused by the vector dimension deviation, and finally obtaining a quantified matching similarity.

[0117] Step S4024: Determine the semantic alignment deviation based on the matching similarity.

[0118] Semantic alignment bias can be understood as the degree of inconsistency between text information and image features at the semantic level.

[0119] Specifically, the matching similarity obtained in step S4023 is compared with a preset similarity threshold, and the difference between the similarity and the threshold is converted into semantic alignment deviation.

[0120] Step S4025: Spatial matching calibration is performed based on the text location information in the target information and the image region location to obtain the first bounding box coordinates corresponding to the text information and the second bounding box coordinates of the image region.

[0121] Image region location can be understood as the specific location of an image region in a document (such as coordinate range or region).

[0122] Text location information can be understood as the specific location of text information in a document (such as coordinate range or region).

[0123] The first bounding box coordinates can be understood as the bounding box coordinates of the text information in the document after spatial matching calibration.

[0124] The second bounding box coordinates can be understood as the bounding box coordinates of the image region in the document after spatial matching calibration.

[0125] Specifically, the text location information and image region location are extracted from the target information, and their coordinate systems are unified; a spatial calibration algorithm is used to correct the coordinate deviation between the text location and the image region location, and spatial matching calibration is performed; after calibration, the coordinates of the first bounding box corresponding to the text information and the coordinates of the second bounding box corresponding to the image region are output respectively.

[0126] Step S4026: Determine the intersection-union ratio based on the coordinates of the first bounding box and the second bounding box.

[0127] Specifically, the intersection-union ratio (IURR) calculation algorithm is used to calculate the intersection area (area of ​​overlapping region) and union area (total area covered by the two bounding boxes) of the two bounding boxes respectively. The quantified IURR is obtained by using the formula IURR = intersection area / union area.

[0128] Step S4027: Determine the spatial alignment deviation based on the cross-union ratio and the cross-union ratio threshold.

[0129] The intersection-union ratio threshold can be understood as a preset critical value used to determine the degree of spatial alignment between text and image regions.

[0130] Spatial alignment deviation can be understood as the degree of inconsistency between text information and image regions in terms of spatial location.

[0131] Specifically, the deviation calculation logic is used to convert the difference between the intersection-union ratio and the threshold into a spatial alignment deviation, so as to obtain the spatial alignment deviation.

[0132] Step S4028: Determine the time difference between the document generation time and the upload time corresponding to the target information.

[0133] Extract the document generation time and upload time from the target information, unify the time formats of the two, use time difference calculation logic to calculate the absolute difference between the two times, determine the specific value of the time difference, and obtain the time difference.

[0134] Step S4029: Determine the time alignment deviation based on the time difference and the time difference threshold.

[0135] Time alignment discrepancy can be understood as the degree of inconsistency between the document generation time and the upload time at the temporal level.

[0136] Specifically, the time difference threshold specified in the preset rule constraints is retrieved, and the deviation calculation logic is used to convert the difference between the time difference and the threshold into a time alignment deviation within the range of 0-1.

[0137] Step S4030: Based on semantic alignment deviation, spatial alignment deviation and temporal alignment deviation, form the alignment deviation result corresponding to the target information.

[0138] The semantic alignment deviation, spatial alignment deviation, and temporal alignment deviation obtained in steps S4024, S4027, and S4029 are retrieved. The weight configuration of the three types of deviations in the preset rule constraints is used to calculate the total alignment deviation value of the target information using a weighted summation algorithm. The total deviation value and the deviation values ​​of each item are integrated to form a complete alignment deviation result.

[0139] The document recognition method provided by this invention obtains a first vector code by vector encoding text information and a second vector code by vector encoding text descriptions generated from image regions corresponding to image features. Based on these two types of vector codes, matching similarity is determined, thereby accurately quantifying semantic alignment deviation and ensuring semantic consistency. Furthermore, by obtaining the coordinates of the first bounding box corresponding to the text information and the coordinates of the second bounding box corresponding to the image region, the intersection-union ratio (IU / R) of the two is calculated, and a IU / R threshold is used to determine spatial alignment deviation, ensuring spatial matching accuracy. Moreover, by determining the time difference between the document generation time and the upload time, a time difference threshold is used to quantify temporal alignment deviation. Finally, the three types of deviations are integrated to form a complete alignment deviation result. Based on the semantic alignment deviation, spatial alignment deviation, and temporal alignment deviation, the accuracy of document recognition can be improved.

[0140] Based on the alignment deviation result in step S103 of this embodiment, the multimodal features corresponding to the target information are determined using the schemes in steps b1 to b3: Step b1 converts the alignment deviation results into constraint information for feature fusion.

[0141] Retrieve the alignment deviation results obtained in step S4030, extract the total alignment deviation value, the deviation values ​​of each item, and the annotation information of deviations exceeding the threshold; refer to the weight configuration in the preset rule constraints, perform a standardization transformation on the alignment deviation results (map the deviation values ​​to the 0-1 standardization range), and combine the deviation priority (the larger the deviation and the higher the weight of the item, the higher the constraint priority) to generate constraint information for feature fusion.

[0142] Step b2: Adjust the attention weights based on the constraint information to obtain the target attention weights.

[0143] Attention weights can be understood as the weights assigned to different types of features (text features, image features) and different biases during the feature fusion process.

[0144] Specifically, the initial attention weights are dynamically adjusted based on the deviation priority in the constraint information. For features and sub-items with larger deviations and higher constraint priority, their attention weights are increased; for features and sub-items with smaller deviations and lower constraint priority, their attention weights are decreased. After adjustment, the sum of the weights is checked to ensure it equals 1, thus obtaining the final target attention weights.

[0145] Step b3: Based on the target attention weight, perform feature fusion on the target information to obtain the multimodal features corresponding to the target information.

[0146] The target attention weights obtained in step b2 are retrieved, and various features (text features, image features, temporal features, etc.) are extracted from the target information. These features are then standardized and preprocessed. Based on the target attention weights, a feature fusion algorithm is used to weight and fuse the various features. Features with high target attention weights have a higher weighting in the fusion process, highlighting their feature information. After fusion, the fusion result is normalized to obtain the multimodal features corresponding to the target information.

[0147] The document recognition method provided by this invention determines the evidence strength information of document data based on multimodal features, and then determines the risk information of document data based on the alignment deviation results. It accurately captures the potential deviations in the semantic, spatial and temporal three-dimensional alignment process, and can effectively avoid the deviations caused by identification based on a single criterion, thereby improving the accuracy of document recognition.

[0148] In one possible implementation, the document recognition method described above may include the following steps: semantic alignment, spatial alignment, and temporal alignment. The specific implementation process is as follows: The semantic alignment function compares the vector of a text sentence with the vector of a text describing an image region.

[0149] Specific implementation: Text embedding: The text segments extracted by OCR are encoded into sentence vectors T_emb using a pre-trained BERT model, with dimensions [n_text, 768], where n_text is the number of text segments. Image region description generation: For each image region (such as a stamp region or a signature region), a text description is generated using a visual description model, also encoded as a vector I_desc with dimensions [n_image, 768]. Similarity calculation: Calculate the cosine similarity matrix between T_emb and I_desc: CosSim[i,j] = cosine_similarity(T_emb[i], I_desc[j]) Bias quantification: If CosSim[i,j]>=0.8, mark it as a match. Rule-related tokens are prioritized for matching: identify rule-related text tokens (such as keywords like amount, date, signature, etc.).

[0150] For matching of rule-related tokens, increase the weight: λ_sem_rule=λ_sem×(1-rule_priority_weight), where rule_priority_weight is determined according to the rule weight and has a value range of [0.1,0.3].

[0151] Experimental comparisons showed that when the cosine similarity was ≥0.8, the semantic matching accuracy reached over 95%. This threshold was validated on 1000 document samples, with a false match rate of <5%.

[0152] Output: Semantic alignment bias matrix λ_sem, dimension [n_text, n_image].

[0153] The spatial alignment function can be used to compare the coordinates of the text box extracted by OCR with the image region.

[0154] Specific implementation: Text box coordinate extraction: OCR outputs the bounding box coordinates of each text line: bbox_text=[x1,y1,x2,y2]; An object detection model (such as YOLO) is used to detect key regions (stamps, signatures, tables, etc.) in the image, obtaining the bounding box bbox_image=[x1,y1,x2,y2]. Calculate the intersection-union ratio (IoU) between the text box and the image region. If IoU >= 0.6, mark it as a match.

[0155] For the signature area, additional checks are performed on the reasonableness of its placement. The IoU and placement scores are combined.

[0156] Threshold selection criteria: IoU ≥ 0.6 allows for tolerance of certain OCR positioning errors. This threshold was validated on 500 document samples, achieving a spatial alignment accuracy of over 92%.

[0157] Output: Spatial alignment bias matrix λ_spa, dimensions [n_text, n_image] The time alignment function compares timestamps or batch sequence numbers to ensure the temporal logic of the evidence chain.

[0158] Specific implementation: Extract the upload time t_upload from file metadata; extract the document generation time t_doc from OCR text (by time pattern matching); extract the related event time t_event from the business system. Calculate the time difference to obtain the time alignment deviation.

[0159] Alignment results summary The outputs of the three aligned branches are combined as follows: Alignment deviation vector: λ=[λ_sem_avg,λ_spa_avg,λ_tmp_avg] Loss composition: L_align=α×λ_sem_avg+β×λ_spa_avg+γ×λ_tmp_avg, where α, β, and γ are weight coefficients with values ​​ranging from [0.2, 0.4], λ_sem_avg is the semantic alignment bias, λ_spa_avg is the spatial alignment bias, and λ_tmp_avg is the temporal alignment bias.

[0160] Audit trail update: Alignment deviations are recorded in audit trail A. Weight coefficients are optimized on the validation set through grid search and cross-validation. Experiments show that the alignment accuracy is highest when α=0.3, β=0.3, and γ=0.2.

[0161] In one possible implementation, for cross-modal attention fusion networks, combined with Figure 4 As shown, it can specifically include: Input layer: Text processing: The text token sequence T is converted into T_emb through the Embedding layer, with dimensions [50, 768] (fixed to 50 tokens, padded with zeros if insufficient, truncated if excessive). Image processing: Image feature map I is converted into I_feat through CNN feature extraction, with dimensions [H×W, 768], where H×W is the feature map space size; Alignment deviation λ is converted into a deviation mask: mask_sem=1-λ_sem_avg (semantic mask, the larger the value, the better the semantic alignment); mask_spa=1-λ_spa_avg(spatial mask); mask_tmp=1-λ_tmp_avg(time mask); The mask value ranges from [0,1], with a larger value indicating better alignment quality; Cross-Attention Layer 1: Set Q=T_emb,K=I_feat,V=I_feat; Application rule mask: scores_masked = Attention_scores × mask_sem × mask_spa × mask_tmp; Calculate cross-attention: Attention = Softmax(scores_masked) × V; Output: H_cross1[50,768]; (3) Cross-Attention Layer 2: Set Q=H_cross1, K=I_feat, V=I_feat; Repeat the masking process described above to perform deep fusion; Output: H_cross[50,768]; (4) Self-Attention Layer (in parallel with cross-attention): Text self-attention: H_self_text = SelfAttention(T_emb)[50,768]; Image self-attention: H_self_image = SelfAttention(I_feat)[H×W,768]; Image feature pooling: H_self_image_pooled=AdaptiveAvgPool2d(H_self_image)[50,768]; (Pool the image feature maps to the same number of text tokens to facilitate subsequent fusion) Concatenation: H_self=Concat(H_self_text,H_self_image_pooled)[100,768]; FusionLayer: Feature concatenation: H_concat=Concat(H_cross,H_self), dimension [150,768]; Fully connected dimensionality reduction: H = FC(H_concat), dimension [50, 768]; FC layer parameters: Input dimensions: 150 × 768 = 115200; Output dimensions: 50 × 768 = 38400; Activation function: ReLU; Weight initialization: Xavier initialization; Function: Unifies feature representation through linear transformation, unifying it into a fixed dimension for easier subsequent processing; Dimensional consistency guarantee: By using adaptive pooling and a fixed number of tokens, we ensure that all intermediate features maintain dimensionality consistency. The dimensionality reduction operation of the FC layer has been experimentally verified to not significantly lose important information (information retention rate >95%). Modal weight calculation: Calculate the sum of the weights of each modality in cross-attention: weight_text=sum(Attention_weights[:,text_indices]); weight_image=sum(Attention_weights[:,image_indices]); weight_meta=sum(Attention_weights[:,metadata_indices]); Normalization: Modal_Weights=[weight_text,weight_image,weight_meta]; Output: F = Softmax(Modal_Weights) = [f_text, f_image, f_meta]; Loss function calculation: L=L_base+α×λ_sem+β×λ_spa+γ×λ_tmp; Where L_base is the base loss for the classification / regression task, and α, β, and γ are the rule weight coefficients (0.2-0.4).

[0162] Technical contributions of loss functions: 1. Loss function design: L_total=L_task+α×L_sem+β×L_spa+γ×L_tmp; in: L_task: The loss of the main task (such as classification cross-entropy), on the order of approximately 0.5-2.0; L_sem=mean(λ_sem): semantic alignment bias, on the order of 0-1; L_spa=mean(λ_spa): Spatial alignment deviation, on the order of 0-1; L_tmp = mean(λ_tmp): Time alignment deviation, on the order of 0-1; α, β, γ: Weighting coefficients, with values ​​ranging from 0.2 to 0.4, to ensure scale matching for each loss term.

[0163] In a specific implementation scenario, this embodiment takes the evidence chain of loan contracts and bank transfer receipts in debt collection business as an example to illustrate the complete process from inputting evidence documents to outputting a credibility profile.

[0164] Evaluation indicators: accuracy rate of evidence strength determination, amount of manual review, and processing time.

[0165] The following evidence documents are received during the input phase: Document 1: Scanned copy of the loan agreement (PDF, 3 pages); Debtor: Zhang San, ID number: 11010119900101XXXX; Loan amount: RMB 100,000; Signature location: bbox=[1200, 800, 1400, 1000]; Contract date: 2025-01-10, Upload time: 2025-01-15 14:30:00; Document 2: Bank transfer receipt (JPG); Transfer amount: 100,000.00 yuan; Transfer time: 2025-01-15 14:30:00; Upload time: 2025-01-15 14:35:00; Preprocessing and rule formalization stages; After processing, the key information extracted is shown in Table 1: Table 1 The rule formalization engine transforms business rules into computable functions: Rule 1: Signature position verification, IoU ≥ 0.75, weight w1 = 0.3; Rule 2: Verify consistency of amount; if the difference is ≤ 1 yuan, the weight w2 = 0.25. Rule 3: Time chain verification, upload time - return time ≤ 5 minutes, weight w3 = 0.2; 3D alignment calculation stage: The 3D alignment module performs alignment verification on the extracted information, and the results are shown in Table 2: Table 2 Alignment results: All alignment deviations are 0, and the rule satisfaction is 1.0.

[0166] Cross-modal attention fusion stage The converged network processing procedure is as follows: Cross-attention layer: Cross-attention calculation is performed between the text token sequence and the image feature map. An alignment bias mask (mask_sem=1.0, mask_spa=1.0, mask_tmp=1.0) is applied to output the preliminary fusion feature H_cross.

[0167] Self-attention layer: Performs self-attention calculation on H_cross and outputs H_self.

[0168] Fusion computation layer: H_cross and H_self are concatenated and then dimensionality reduced by a fully connected layer to obtain a unified fusion feature vector H (768 dimensions).

[0169] Based on the fusion feature H and the alignment results, a credibility profile is calculated, as shown in Table 3: Table 3 Strategy triggering and output: Calculation process: U_mean=(mean([0.92,0.98,0.95,0.88,0.02,0.96])×0.5)-(max([0.02,0.05,0.03])×0.3)+(min([0.45,0.50,0.05])×0.2); U_mean=(0.785×0.5)-(0.05×0.3)+(0.05×0.2)=0.3925-0.015+0.01=0.3875; In actual calculations, the formula for calculating U_mean needs to consider more factors; this is just an example.

[0170] Combination Figure 5 As shown, the policy trigger judgment may include: U_mean=0.82>0.7 (satisfies the condition); R_max = 0.05 < 0.6 (condition met); F_min=0.05<0.1 (Metadata weight is too low); Triggering result: Downgrade processing (marks metadata requiring manual review, but allows access to subsequent systems).

[0171] In some embodiments, the document recognition device of the present invention can be implemented in a combination of hardware and software. As an example, the document recognition device of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the document recognition method of the present invention. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0172] The modules described in the embodiments of this invention can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.

[0173] An electronic device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-mentioned document recognition methods. That is, an electronic device according to an embodiment of the present invention may include, but is not limited to: a processor and a memory; the memory is used to store the computer program; the processor is used to execute the document recognition method shown in any embodiment of the present invention by calling the computer program.

[0174] In one alternative embodiment, an electronic device is provided, such as Figure 6 As shown, Figure 6 The illustrated electronic device 6000 includes a processor 6001 and a memory 6003. The processor 6001 and the memory 6003 are connected, for example, via a bus 6002. Optionally, the electronic device 6000 may further include a transceiver 6004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 6004 is not limited to one type, and the structure of the electronic device 6000 does not constitute a limitation on the embodiments of the present invention.

[0175] Processor 6001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 6001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0176] Bus 6002 may include a path for transmitting information between the aforementioned components. Bus 6002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 6002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The bus 6002 is represented by only one thick line, but this does not mean that there is only one bus or one type of bus.

[0177] The memory 6003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0178] The memory 6003 stores application code (computer program) for executing the present invention, and its execution is controlled by the processor 6001. The processor 6001 executes the application code stored in the memory 6003 to implement the content shown in the foregoing method embodiments.

[0179] Among them, electronic devices can also be terminal devices, which can be any device that can install applications, including at least one of smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, smart TVs, and smart in-vehicle devices.

[0180] It should be noted that, Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0181] An embodiment of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-described document recognition methods.

[0182] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device, etc.

[0183] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the document recognition method described above.

[0184] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0185] It should be understood that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0186] The computer-readable storage medium provided in this invention can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EEPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0187] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.

[0188] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

[0189] It should be noted that the terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and represent a limitation on a specific order or sequence. Where appropriate, the order of use for similar objects can be interchanged so that the embodiments of this application described herein can be implemented in an order other than that shown or described.

[0190] Those skilled in the art will recognize that this invention can be implemented as a system, method, or computer program product. Therefore, this invention can be specifically implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a circuit, module, or system. Furthermore, in some embodiments, this invention can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0191] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A document recognition method, characterized in that, The method includes: Extract target information from multi-source heterogeneous document data; wherein the target information includes at least one of text information, image features, location information, and temporal information; Based on preset rule constraints, the target information is aligned to obtain the alignment deviation result corresponding to the target information; Based on the alignment deviation results, the multimodal features corresponding to the target information are determined; The document data is identified based on the alignment deviation results and the multimodal features.

2. The document recognition method according to claim 1, characterized in that, The extraction of target information from multi-source heterogeneous document data includes: The multi-source heterogeneous document data is extracted and processed to obtain text confidence and information to be processed; wherein, the information to be processed includes: text information to be processed, image information to be processed, location information to be processed, and time sequence information to be processed. Based on the text confidence level, anomaly detection processing is performed on the information to be processed to obtain the target information; The anomaly detection process includes: When the text recognition confidence level is lower than the preset confidence threshold or the text information to be processed is invalid, the image information to be processed is determined as the target information; When the preset rule constraints are abnormal, the information to be processed will be identified as the target information.

3. The document recognition method according to claim 1, characterized in that, The method further includes: The business rules to be processed are converted into computable constraints, and judgment thresholds and weights are configured for the computable constraints to form a set of rule constraints. The preset rule constraints are determined based on the predefined rule template, historical document data, and the set of rule constraints.

4. The document recognition method according to claim 3, characterized in that, The process of determining the preset rule constraints based on predefined rule templates, historical document data, and the set of rule constraints includes: Based on the predefined rule template and historical document data, the judgment threshold and weight of each rule constraint in the rule constraint set are optimized to obtain the target rule constraint set; Detect whether there are logically conflicting rule constraints in the target rule constraint set; If there are logically conflicting rule constraints in the target rule constraint set, sort them according to their weights to obtain the sorting result; Based on the sorting results, determine the rule satisfaction of rule constraints with logical conflicts; Based on the aforementioned rule satisfaction, rule constraints for alignment processing are determined from rule constraints with logical conflicts. Based on the rule constraints used for alignment processing and the rule constraints that do not have logical conflicts, the preset rule constraints are determined.

5. The document recognition method according to claim 1, characterized in that, Based on preset rule constraints, the target information is aligned to obtain the alignment deviation result corresponding to the target information, including: Based on preset rule constraints, the text information is vector-encoded to obtain the first vector code; A text description is generated for the image region corresponding to the image features, and the text description is vector-encoded to obtain a second vector code; Based on the first vector encoding and the second vector encoding, the matching similarity is determined; Based on the matching similarity, the semantic alignment deviation is determined; Spatial matching and calibration are performed based on the text location information in the target information and the image region location to obtain the first bounding box coordinates corresponding to the text information and the second bounding box coordinates of the image region. Determine the intersection-union ratio based on the coordinates of the first and second bounding boxes; Based on the cross-union ratio and the cross-union ratio threshold, the spatial alignment deviation is determined; Determine the time difference between the document generation time and upload time corresponding to the target information; Based on the time difference and the time difference threshold, the time alignment deviation is determined; Based on the semantic alignment deviation, the spatial alignment deviation, and the temporal alignment deviation, an alignment deviation result corresponding to the target information is formed.

6. The document recognition method according to claim 1, characterized in that, The step of determining the multimodal features corresponding to the target information based on the alignment deviation result includes: The alignment deviation results are converted into constraint information for feature fusion; The attention weights are adjusted based on the constraint information to obtain the target attention weights; Based on the target attention weight, feature fusion is performed on the target information to obtain the multimodal features corresponding to the target information.

7. The document recognition method according to claim 1, characterized in that, Based on the alignment deviation results and the multimodal features, the document data is identified, including: Based on the multimodal features, determine the evidence strength information of the document data; Based on the alignment deviation results, the risk information of the document data is determined; The document data is identified based on the evidence strength information and the risk information.

8. A document recognition device, characterized in that, The device includes: The first processing module is used to extract target information from multi-source heterogeneous document data; wherein the target information includes at least one of text information, image features, location information, and temporal information; The second processing module is used to perform alignment processing on the target information based on preset rule constraints to obtain the alignment deviation result corresponding to the target information; The determination module is used to determine the multimodal features corresponding to the target information based on the alignment deviation result; The identification module is used to identify the document data based on the alignment deviation result and the multimodal features.

9. An electronic device, characterized in that, include: A processor and a memory connected to the processor; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to cause the processor to perform the document recognition method as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the document recognition method according to any one of claims 1-7.

Images