Contract document analysis method and system based on multi-modal adaptive feature fusion

Abstract

The application discloses a contract document analysis method and system based on multi-modal adaptive feature fusion. The method comprises the following steps: obtaining an image and text data of a contract document and performing preprocessing; performing multi-modal feature extraction from the preprocessed image and contract text to obtain image features and text features; calculating the semantic correlation of the image features and the text features and performing adaptive weight fusion to obtain fused features; capturing the context correlation of the contract text and optimizing the position and logical consistency of the initial bounding box according to the fused features to obtain a prediction result; and binding the prediction result with the image features and the text features and outputting a structured analysis report. The method of the application can fully meet the high accuracy and traceability required by the contract document analysis based on multi-modal adaptive feature fusion, and solve the technical problems of multi-modal feature fusion, bounding box generation accuracy and analysis process transparency in the prior art.

Description

Technical Field

[0001] This invention relates to document processing methods, and more specifically to a contract document parsing method and system based on multimodal adaptive feature fusion. Background Technology

[0002] As core documents in legal affairs, contracts embody the rights and obligations of all parties. Accurate interpretation of these documents is crucial for ensuring a clear understanding and effective execution of contract terms, as well as providing strong evidence in case of disputes. With the acceleration of digital transformation, more and more enterprises and organizations are seeking to improve the efficiency and accuracy of contract management through automation tools. However, in this process, how to efficiently and accurately extract key information from contract documents has become a pressing issue.

[0003] Currently, attempts to integrate text and image information employ fixed weights or simple weighted average strategies. This approach fails to adequately account for the differences in semantic relevance between different contract clauses. For example, in some clauses such as "Description of the Subject Matter," text may be the primary content carrier; while in others, such as "Equipment Diagram," images may carry more crucial information. Furthermore, existing methods struggle to dynamically adjust feature weights based on changes in content complexity, resulting in poor performance when handling structurally complex legal provisions or visually significant portions such as charts and graphs.

[0004] Existing technologies primarily rely on local visual features to generate bounding box information such as text boxes or table boxes, neglecting the importance of textual context and semantics. Specifically, certain parts of a contract document may have cross-regional semantic relationships, but existing technologies cannot identify and utilize such hierarchical relationships to optimize bounding box generation. Furthermore, when performing bounding box merging or splitting operations, there is a lack of verification mechanisms to ensure the merged result conforms to textual logic consistency, easily leading to erroneous merging, such as inappropriately combining "Party A's obligations" with "Party B's rights."

[0005] Existing systems, while outputting bounding box location and category predictions, do not provide sufficient explanations of the parsing process. This makes it difficult for reviewers to understand the system's decision-making rationale during the review process. This is particularly problematic for legal documents, which require a high degree of reliability and traceability. However, current methods often overlook this, failing to connect key features influencing decisions, such as specific keywords or table structures, nor clearly explaining the logical rules behind bounding box merging or segmentation—whether based on semantic relevance, visual continuity, or other criteria. This situation severely impacts work efficiency and reduces user trust in the system's decisions.

[0006] Therefore, it is necessary to design a new method to achieve the high accuracy and traceability required for contract document parsing based on multimodal adaptive feature fusion, and to solve the technical problems of existing technologies in terms of multimodal feature fusion, bounding box generation accuracy, and transparency of the parsing process. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and system for parsing contract documents based on multimodal adaptive feature fusion.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: a contract document parsing method based on multimodal adaptive feature fusion, comprising:

[0009] The system acquires the image and text data of the contract document and performs preprocessing to obtain the preprocessed image and contract text.

[0010] Multimodal feature extraction is performed on the preprocessed image and contract text to obtain image features and text features;

[0011] The semantic correlation between the image features and the text features is calculated, and adaptive weight fusion is performed to obtain fused features;

[0012] The contextual relevance of the contract text is captured, and the position and logical consistency of the initial bounding box are optimized based on the fusion features to obtain the prediction result;

[0013] The prediction results are bound to the image features and text features to output a structured analysis report.

[0014] The further technical solution is as follows: the acquisition of image and text data of the contract document, and the preprocessing to obtain preprocessed image and contract text, includes:

[0015] The process involves acquiring image and text data of the contract document, performing resolution normalization and tilt correction on the image, and performing OCR recognition and cleaning on the text data to remove noisy characters, thereby obtaining a structured text sequence and a preprocessed image and contract text.

[0016] The further technical solution is as follows: Multimodal feature extraction is performed from the preprocessed image and contract text to obtain image features and text features, including:

[0017] A fine-tuned BERT model is used to semantically encode the preprocessed contract text to generate text features.

[0018] Multi-scale visual features are extracted from the preprocessed image using an FPN network to obtain image features, which include font style, layout structure, and key information such as table borders.

[0019] A further technical solution is as follows: The step of calculating the semantic correlation between the image features and the text features, and performing adaptive weight fusion to obtain fused features, includes:

[0020] The interaction features between the image features and the text features are calculated using a multi-head attention mechanism to determine the similarity between the image features and the text features;

[0021] Calculate the text complexity and image complexity separately;

[0022] Calculate dynamic weights based on the text complexity, the image complexity, and the similarity.

[0023] By combining dynamic weights, the image features and the text features are fused into a fused feature.

[0024] The further technical solution is as follows: the calculation of text complexity and image complexity respectively includes:

[0025] The evaluation is performed by calculating a weighted sum of sentence length, legal term density, and syntax tree depth. The legal term density is based on the matching results of a preset legal term database, while the syntax tree depth is based on the number of nesting levels identified by the grammar rules.

[0026] Image complexity is obtained by weighting the edge detection score, color channel variance, and the number of object detections.

[0027] The further technical solution is as follows: capturing the contextual relevance of the contract text and optimizing the position and logical consistency of the initial bounding box based on the fusion features to obtain the prediction result includes:

[0028] A bidirectional LSTM is used to construct a context encoder, which captures the semantic association features between clauses based on each text feature to obtain context features;

[0029] Candidate bounding boxes are generated based on the layout detection model, and the context features are used to perform semantic filtering and position fine-tuning on the candidate bounding boxes to obtain the prediction results.

[0030] The further technical solution is as follows: binding the prediction result with the image features and text features, and outputting a structured analysis report, includes:

[0031] The prediction results are bound to the image features and text features to form a traceable analysis result, providing a detailed analysis decision path, including key thresholds, triggering keywords and visual features for merging / splitting operations, to obtain a structured analysis report, and the structured analysis report is output.

[0032] This invention also provides a contract document parsing system based on multimodal adaptive feature fusion, comprising:

[0033] The acquisition unit is used to acquire the image and text data of the contract document and perform preprocessing to obtain the preprocessed image and contract text.

[0034] The feature extraction unit is used to extract multimodal features from the preprocessed image and contract text to obtain image features and text features;

[0035] The fusion unit is used to calculate the semantic correlation between the image features and the text features, and to perform adaptive weight fusion to obtain fused features;

[0036] An optimization unit is used to capture the contextual relevance of the contract text and optimize the position and logical consistency of the initial bounding box based on the fusion features to obtain the prediction result;

[0037] The binding output unit is used to bind the prediction result with the image features and text features, and output a structured parsing report.

[0038] The present invention also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the above-described method.

[0039] The present invention also provides a storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0040] The advantages of this invention compared to existing technologies are as follows: By acquiring and preprocessing image and text data of contract documents, this invention achieves efficient extraction and fusion of multimodal information. First, advanced encoding techniques are used to extract features from both images and text, and the semantic relevance between them is calculated to dynamically adjust weights for adaptive fusion, ensuring that different information sources are appropriately represented according to their importance during the parsing process. Next, by capturing the contextual relevance of the contract text, the position and logical consistency of bounding boxes are optimized, significantly improving the accuracy of entity recognition and classification. Finally, the prediction results are bound to the original features to form a structured parsing report, providing detailed decision-making basis and explanation chains. This not only improves the overall accuracy of contract document parsing based on multimodal adaptive feature fusion but also enhances the transparency and traceability of the entire process, effectively addressing the shortcomings of existing technologies in multimodal feature fusion, bounding box generation accuracy, and parsing process transparency.

[0041] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A schematic diagram illustrating an application scenario of the contract document parsing method based on multimodal adaptive feature fusion provided in this embodiment of the invention;

[0044] Figure 2 A flowchart illustrating the contract document parsing method based on multimodal adaptive feature fusion provided in an embodiment of the present invention;

[0045] Figure 3 for Figure 2 A schematic diagram of the sub-process of step S120;

[0046] Figure 4 for Figure 2 A schematic diagram of the sub-process of step S130;

[0047] Figure 5 for Figure 4 A schematic diagram of the sub-process of step S132;

[0048] Figure 6 for Figure 2 A schematic diagram of the sub-process of step S140;

[0049] Figure 7A schematic block diagram of a contract document parsing system based on multimodal adaptive feature fusion provided in an embodiment of the present invention;

[0050] Figure 8 A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0053] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0054] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0055] Please see Figure 1 and Figure 2 , Figure 1 This is a schematic diagram illustrating an application scenario of the contract document parsing method based on multimodal adaptive feature fusion provided in an embodiment of the present invention. Figure 2This is a schematic flowchart illustrating the contract document parsing method based on multimodal adaptive feature fusion provided in this embodiment of the invention. This method is applied in a server. The server interacts with the terminal, achieving efficient parsing of contract document image and text data by combining multimodal feature extraction, adaptive weight fusion, and contextual relevance capture. First, a fine-tuned BERT model and FPN network are used to extract semantic and visual features from the contract text and image, respectively. Their semantic relevance and complexity are calculated to determine dynamic weights for feature fusion, ensuring the accuracy and completeness of the information. Next, a bidirectional LSTM is used to construct a context encoder to capture the semantic relationships between clauses, optimizing the position and logical consistency of bounding boxes, thus improving the accuracy of bounding box generation. Finally, by binding the prediction results with image and text features, a traceable structured parsing report is formed, enhancing the transparency of the parsing process and the clarity of the decision path. This fully meets the high accuracy and traceability requirements of contract document parsing based on multimodal adaptive feature fusion, addressing the shortcomings of existing technologies in multimodal feature fusion, bounding box generation accuracy, and parsing process transparency.

[0056] Figure 2 This is a flowchart illustrating the contract document parsing method based on multimodal adaptive feature fusion provided in an embodiment of the present invention. Figure 2 As shown, the method includes the following steps S110 to S150.

[0057] S110. Obtain the image and text data of the contract document and perform preprocessing to obtain the preprocessed image and contract text.

[0058] In this embodiment, the preprocessed image refers to an image suitable for multi-scale visual feature extraction obtained after a series of operations. These operations include, but are not limited to, resolution normalization and tilt correction. Resolution normalization ensures that contract images from different sources have a uniform standard size, avoiding inconsistencies in feature extraction caused by differences in the resolution of the original images. Tilt correction is used to correct image tilt caused by improper scanning or shooting angles, ensuring that the horizontal and vertical directions of key elements such as table borders and text rows are correct, thereby improving the accuracy of subsequent candidate bounding boxes generated based on the layout detection model.

[0059] The preprocessed contract text refers to the structured text sequence obtained after OCR (Optical Character Recognition) recognition and cleaning of the original text data. First, OCR technology is used to convert the text content in the contract document into machine-readable text format. Then, a text cleaning process removes noisy characters (such as non-printable characters, garbled text, etc.) and performs necessary formatting adjustments, such as merging consecutive spaces into single spaces and removing redundant line breaks. Ultimately, a clear, coherent, and structured text sequence is obtained, providing high-quality input for further text encoding and semantic analysis.

[0060] Specifically, the image and text data of the contract document are acquired, the image is normalized for resolution and tilted for correction, and the text data is subjected to OCR recognition and cleaning to remove noisy characters, resulting in a structured text sequence, thus obtaining the preprocessed image and contract text.

[0061] Collect relevant image and text information from contract documents in physical or digital form. Adjust image sizes to a uniform standard and correct any tilting issues to ensure accurate display of image content. Use OCR technology to convert text in images into text, and remove unnecessary characters through a cleaning process to ensure the cleanliness and usability of the text data. Further clean up distracting elements in the text data, such as special symbols and incorrect encoding, and format it into an easy-to-understand and process format.

[0062] This processing of images and text not only improves the efficiency and quality of multimodal feature extraction, but also lays a solid foundation for subsequent steps such as constructing context-related capture and optimizing bounding box positions.

[0063] S120. Multimodal feature extraction is performed from the preprocessed image and contract text to obtain image features and text features.

[0064] In this embodiment, image features are a set of features extracted from the contract document image that represent its visual information. These features include not only text content but also non-text elements such as font style (e.g., bold, underline), layout structure (e.g., indentation, line spacing), and table borders. Specifically:

[0065] Font style: Identify the importance or special nature of specific clauses by detecting font attributes in the text area, such as whether it is bold or underlined.

[0066] Layout structure: Analyzing the layout of text paragraphs, such as indentation and line spacing, helps to understand the logical relationships between different clauses.

[0067] Table borders: Identify and capture the boundary information of tables and their internal cells, which is especially important for understanding and parsing complex financial data or lists of items.

[0068] To extract the aforementioned features, a method based on FPN (Feature Pyramid Network) was employed. FPN can effectively extract image features at multiple scales, ensuring accurate recognition even for small fonts or blurry parts.

[0069] Text features are vector representations extracted from the text content of contract documents to represent their semantic information. In this invention, a BERT model finely tuned to adapt to legal terminology and contract terms is used to encode the text, generating a text feature vector T. The specific components of text features include, but are not limited to:

[0070] Semantic encoding: The BERT model transforms each word or phrase into a high-dimensional vector that captures the meaning of the word / phrase in its context.

[0071] Legal terminology density: Given the specialized nature of contract texts, special attention should be paid to the number and distribution of legal terms contained therein, as they are often key to understanding the contract terms.

[0072] Syntactic tree depth: Determining the complexity of sentence structure through syntactic analysis, especially the presence or absence of nested conditional statements, is crucial for understanding the relationships between clauses.

[0073] In one embodiment, please refer to Figure 3 The above-mentioned step S120 may include steps S121 to S122.

[0074] S121. The preprocessed contract text is semantically encoded using a fine-tuned BERT model to generate text features.

[0075] This step utilizes the BERT model, specifically optimized for contract texts, to perform deep learning-driven semantic encoding on each paragraph or clause. The advantage of this approach is that it can more accurately capture subtle differences in contract texts, especially when dealing with highly specialized and semantically complex legal language.

[0076] S122. Using an FPN network, extract multi-scale visual features from the preprocessed image to obtain image features, wherein the image features include font style, layout structure, and key information such as table borders.

[0077] In this step, the FPN network is used to extract rich visual features from the preprocessed contract image. This method can not only recognize clear text content, but also effectively handle low-quality scans such as those that are tilted or blurry. More importantly, it can capture visual cues that are crucial for understanding the contract, such as the use of specific font styles, detailed typographic structure, and precise table layout.

[0078] Through these two steps, the system can comprehensively and meticulously acquire all the important information in the contract document, laying a solid foundation for subsequent steps such as adaptive weight adjustment mechanism, bounding box generation and optimization.

[0079] S130. Calculate the semantic correlation between the image features and the text features, and perform adaptive weight fusion to obtain fused features.

[0080] In this embodiment, the fused feature is a comprehensive feature vector F obtained by combining image features and text features, and using a dynamic weighting method based on their respective complexity and semantic similarity. This feature vector not only reflects the information in the text content but also captures visual cues in the image, providing fundamental support for subsequent bounding box generation, optimization, and result interpretation.

[0081] In one embodiment, please refer to Figure 4 The above step S130 may include steps S131 to S134.

[0082] S131. Calculate the interaction features between the image features and the text features using a multi-head attention mechanism to determine the similarity between the image features and the text features.

[0083] In this embodiment, the interaction feature refers to the matrix obtained after performing interaction analysis on image feature I and text feature T using a multi-head attention mechanism. , where d represents the feature dimension. This mechanism can effectively quantify the strength of the relationship between text and images, that is, the degree of their association in a specific context.

[0084] First, the text feature vector T and the image feature vector I are input into the cross-attention layer, and the interaction feature matrix M between them is calculated using a multi-head attention mechanism. Then, based on this matrix, the cosine similarity sim(T, I) between the two is calculated as an indicator of the degree of their correlation.

[0085] Specifically, first, new feature representations for text and images are extracted from the interaction feature matrix M. This typically means performing appropriate transformations or aggregation operations on the matrix to obtain a comprehensive feature vector for each modality (text and image). For example, dimensionality can be reduced and a single feature vector generated through pooling operations such as average pooling or max pooling.

[0086] For text feature vectors: the text-related parts can be extracted from M and pooling operations can be applied.

[0087] For image feature vectors: similarly, extract the image-related parts from M and apply the appropriate pooling method.

[0088] Next, the two eigenvectors obtained need to be L2 normalized. This means that each vector is divided by its own Euclidean norm (i.e., vector length) to make their length 1. This is done to ensure that the calculated similarity is not affected by the size of the vectors, but only reflects the similarity in direction.

[0089] Finally, cosine similarity is calculated using the standardized feature vectors. Cosine similarity is obtained by dividing the result of the dot product of the two vectors by the magnitude of their product.

[0090] S132. Calculate the text complexity and image complexity respectively.

[0091] In this embodiment, text complexity refers to the difficulty of understanding the text content, and is calculated by weighted summation of three factors: sentence length L (number of tokens), legal terminology density T (number of terms / total number of tokens), and syntax tree depth S (nesting level). The formula is as follows: Text complexity = 0.3 × L + 0.4 × T + 0.3 × S.

[0092] Image complexity refers to the degree of complexity of image information, which includes a weighted combination of edge detection score E, color channel variance C, and the number of object detections O. The formula is as follows: Image complexity = 0.4 × E + 0.2 × C + 0.4 × O.

[0093] To accurately evaluate these parameters, the Canny operator was used for edge detection, the Hough transform was used to identify valid edges, the sum of the standard deviations of the RGB three channels was used to represent the variance of the color channels, and the number of layout elements was counted to evaluate the number of object detections.

[0094] In one embodiment, please refer to Figure 5 The above step S132 may include steps S1321 to S1322.

[0095] S1321. Evaluation is performed by calculating the weighted sum of sentence length, legal term density, and syntax tree depth, where legal term density is based on the matching results of a preset legal term database, and syntax tree depth is based on the number of nesting levels identified by the grammar rules.

[0096] S1322. The image complexity is obtained by weighting the edge detection score, color channel variance, and the number of object detections.

[0097] S133. Calculate dynamic weights based on the text complexity, the image complexity, and the similarity.

[0098] In this embodiment, the dynamic weight α is based on the text complexity. Image complexity Similarity between text and images It is calculated using the following formula: ;in, This is the Sigmoid function (output range 0-1). and For learnable parameters, Let T be the cosine similarity between T and I.

[0099] S134. Combining dynamic weights, the image features and the text features are fused into a fused feature.

[0100] Ultimately, through dynamic weights We obtain the fused feature F by weighted summation of text feature T and image feature I: .

[0101] This method allows the system to automatically adjust the fusion ratio of text and image information based on different types of contract documents, thereby capturing key information more accurately and improving parsing precision and reliability. Simultaneously, this strategy enhances the model's ability to handle low-quality scanned documents, reduces false recognition rates, and improves the accuracy of bounding box localization.

[0102] In this embodiment, S120 quantifies the complexity of text features and image features, and dynamically adjusts their weights during fusion based on these complexities and their semantic similarity. This step is crucial for achieving optimal fusion of text and image features.

[0103] Text complexity is calculated based on three dimensions: sentence length, legal terminology density, and syntax tree depth. Its value ranges from 0 to 1, with higher values ​​indicating more complex text. Sentence length (L) is calculated by counting the total number of tokens in a single sentence. This metric reflects the length of the sentence; longer sentences typically contain more information.

[0104] Legal terminology density calculation (T): First, terms in the sentence are matched using a pre-set legal terminology database. Base density = number of terms / total number of tokens. This metric measures the proportion of technical terms in a sentence; a higher proportion indicates greater difficulty in understanding.

[0105] Syntactic tree depth calculation (S): This metric calculates the number of levels by matching nested markers such as "if," "then," and "and" using grammatical rules. This indicator reflects the complexity of the sentence structure; deeply nested sentence structures are often more difficult to parse.

[0106] Image complexity is evaluated based on three aspects: edge detection results, color channel variance, and number of object detections. Its value is also between 0 and 1, with a higher value indicating a more complex image.

[0107] The Canny operator is used to extract image edges. First, the proportion of total edge pixels is calculated (total edge pixels / total image pixels).

[0108] Then, filter by edge validity: use Hough transform to identify "valid edges" such as table borders and chart axes (edges that meet the requirements of linear continuity and length ≥ 10 pixels), and count the percentage of valid edges (valid edge pixels / total edge pixels).

[0109] The final edge detection score = total edge percentage × effective edge percentage, normalized to the range of 0-1.

[0110] Calculation of color channel variance C:

[0111] Calculate the standard deviation of pixel values ​​for each of the three RGB channels, and sum them to obtain the total channel variance.

[0112] If the image is the signature or seal area in a contract, add a "feature uniqueness coefficient": calculate the color contrast between the area and the surrounding background (contrast = (area mean - background mean) / 255).

[0113] Adjust the variance by formulating the total variance × (1 + contrast × 0.3) and then normalize it by formulating the color complexity score = min(corrected variance / 300, 1) (where 300 is the color complexity threshold for the contract image).

[0114] Calculation of the number of targets detected, O:

[0115] This involves counting the total number of various layout elements (such as text blocks, tables, and graphics) in an image. A higher number of layout elements usually indicates higher image complexity.

[0116] Dynamic weights are determined dynamically based on text complexity, image complexity, and the similarity between text and images, achieving a dynamic balance between text and image features. This strategy not only improves the model's ability to understand different types of contract documents but also enhances its robustness in handling low-quality scanned documents.

[0117] S140. Capture the contextual relevance of the contract text, and optimize the position and logical consistency of the initial bounding box based on the fusion features to obtain the prediction result.

[0118] In this embodiment, the prediction result refers to the generation of an accurate and logically coherent set of bounding boxes and its parsing report by capturing the contextual relevance of the contract text and optimizing the position and logical consistency of the initial bounding boxes based on fusion features.

[0119] In step S140, the main objective is to capture the contextual relevance of the contract text and optimize the position and logical consistency of the initial bounding box based on fused features, thereby obtaining the final prediction result. This process includes two key sub-steps: contextual semantic modeling (S141) and bounding box optimization based on contextual features (S142). Through these two steps, the accuracy of the bounding box and the logical consistency of the parsing results can be effectively improved.

[0120] In one embodiment, please refer to Figure 6 The above step S140 may include steps S141 to S142.

[0121] S141. A bidirectional LSTM is used to construct a context encoder to capture the semantic association features between clauses based on each text feature, so as to obtain context features.

[0122] In this embodiment, contextual features refer to the deep semantic information of each text token obtained by performing bidirectional LSTM processing on the contract text sequence. This feature not only considers the meaning of a single token itself, but also integrates the semantic information of its surrounding context.

[0123] In practice, the contract text is first arranged in the order of reading as the input sequence. Then, a bidirectional LSTM model is used to learn the dependencies between these text tokens. The purpose of this is to capture the complex semantic relationships between clauses, such as the subordinate relationship between "Article 3" and "Section 3.1", and the referential relationship between "Party A" and "Party B".

[0124] S142. Generate candidate bounding boxes based on the layout detection model, and use the context features to perform semantic filtering and position fine-tuning on the candidate bounding boxes to obtain the prediction results.

[0125] The layout detection model is used to identify various layout elements (such as text, tables, etc.) in the contract, and a candidate bounding box containing coordinate information (x1, y1, x2, y2) is generated for each element.

[0126] For each candidate bounding box, the text features are compared with the global context features, and the cosine similarity between them is calculated. This step aims to evaluate whether the content within the bounding box is consistent with the overall document context.

[0127] The bounding boxes are merged or split based on their similarity scores. For example, if a bounding box contains unrelated topics (such as "Party A's obligations" and "Party B's rights"), it should be split into several smaller boxes; conversely, if two adjacent boxes contain highly related content, they can be merged into a larger box.

[0128] Position fine-tuning: Based on contextual features, the required offset Δ(x1, y1, x2, y2) is predicted, and the specific coordinate correction is implemented through a fully connected layer structure. During this process, considering the differences in text density in different regions, the dynamic adjustment coefficient α can vary between 0.1 and 0.3 to ensure that the offset is adapted to specific text layout features.

[0129] After the optimization steps described above, the system can output a more accurate and logically coherent set of bounding boxes. These bounding boxes are not only accurately located, but also more semantically reasonable, meeting the stringent requirements for document parsing results in legal scenarios.

[0130] Additionally, the category label of each bounding box (such as "liability for breach of contract") can be linked to its corresponding original features (keywords, visual styles, etc.) to form an easy-to-understand and traceable analysis report.

[0131] In this embodiment, step S140 first uses bidirectional LSTM technology to create a context encoder. This encoder receives a sequence of contract text arranged in reading order as input and outputs the context feature C=[C1, C2, ..., C] corresponding to each text token. n These contextual features include not only the meaning of a single token, but also the dependencies and semantic associations in its surrounding context, such as the semantic association features between clauses (e.g., the co-occurrence probability of "Article 3" and "Section 3.1", the referential relationship between "Party A" and "Party B").

[0132] Candidate bounding boxes are generated based on the layout detection model. Each box contains coordinates (x1, y1, x2, y2) as well as an initial class and confidence score.

[0133] The cosine similarity between the text features within each candidate bounding box and the global context features is calculated. Semantic fragmentation boxes are then split or merged to ensure that the text within each box belongs to the same logical unit. Simultaneously, a semantic association graph is used to determine whether multiple citations belong to the same semantic unit, thus avoiding erroneous splitting.

[0134] Position fine-tuning: The required offset Δ(x1, y1, x2, y2) of the bounding box is predicted using context features, and the position of the bounding box is dynamically adjusted by a fully connected layer based on the prediction function f(C, Box_init)×α of the context features C and the initial box parameter Box_init, ensuring that the adjusted box is adapted to the text layout features.

[0135] S150. Bind the prediction results with the image features and text features, and output a structured analysis report.

[0136] In this embodiment, the prediction results are bound with the image features and text features to form a traceable analysis result, providing a detailed analysis decision path, including key thresholds, triggering keywords and visual features for merging / splitting operations, to obtain a structured analysis report, and the structured analysis report is output.

[0137] The final predicted category is bound to key features, providing both textual and image-based evidence. Key nodes for bounding box merging / segmentation are visualized through the decision path, enhancing the transparency and interpretability of the parsing results. This process helps improve the accuracy and reliability of document parsing, meeting the needs of complex contract text parsing.

[0138] By combining the prediction results with the image features, text features, and other three types of features, a result format is formed that can track each analytical step and the basis for decision-making. This not only helps in understanding the final analytical conclusions but also provides a foundation for possible future reviews or further analysis.

[0139] The analysis report should include the following key sections to ensure its comprehensiveness and traceability:

[0140] Key thresholds for merge / split operations: Clearly specify the specific criteria used when performing bounding box merges or splits, such as similarity thresholds (e.g., 0.8 indicates that two text blocks belong to the same semantic unit), how these thresholds are determined, and the logic behind them.

[0141] Triggering keywords and visual features: Record keywords or visual elements that lead to specific decisions (such as merging / splitting). For example, when "liability for breach of contract" and "compensation for losses" appear in the same text block, and the font is consistent and the line spacing is less than 5px, the system may decide to treat them as a whole.

[0142] Other important decision points: In addition to the two points mentioned above, any other decision factors should also be included, such as judging whether the strength of the relationship between the clauses is sufficient to keep them within the same bounding box based on the semantic association graph.

[0143] Finally, based on all the above information, a structured and easy-to-understand analysis report is generated. This report not only summarizes the entire document analysis process and results but also details the reasons and basis behind each decision, thus providing a comprehensive and in-depth understanding. Such a report is extremely valuable for subsequent review, legal consultation, or use as evidence.

[0144] In summary, step S150 emphasizes building a transparent and verifiable analytical framework by meticulously integrating multimodal data (images + text) with model predictions, providing users with value beyond simple results.

[0145] In processing documents, this embodiment dynamically adjusts the weights of text and images during feature fusion by quantifying their "semantic relevance" and "content complexity." Specifically, when textual information (such as legal clauses) is crucial for understanding the entire document, textual features are given higher weight (α close to 1). Conversely, if image elements (such as tables or signatures) are more critical, the text weight is reduced (α close to 0).

[0146] To improve parsing accuracy, the system utilizes the semantic relationships between text contexts to optimize the initially identified bounding boxes. For example, if the semantic similarity of text within two adjacent boxes exceeds 0.85, they are considered to belong to the same semantic unit and are merged. Conversely, if the text within a box deviates from the overall context theme by more than 0.7, it is considered a false detection and is deleted.

[0147] To enhance the transparency and traceability of the results, the system introduces a "feature tracing" mechanism. This mechanism binds the final parsed result (such as "liability clauses for breach of contract") with the original input features (such as the triggering keyword "penalty" or the visual feature "bold title"), and records the rules used in the decision-making process (such as "merging threshold 0.85"), thus forming a complete explanation chain.

[0148] Compared to fixed-weight methods, the model improved the F1 score by 18%-25% when processing complex contract clauses; for low-quality scanned documents, the error recognition rate was reduced by 30%. The cross-clause bounding box error merging rate was reduced by 40%, and the bounding box localization accuracy for small-font clauses was improved by 22%. Manual review efficiency was improved by 50%, feature-level interpretation was provided, the compliance requirements for parsing results in the financial and legal fields were strengthened, and the legal risks caused by misinterpretation were reduced.

[0149] Through the steps and techniques described above, the method of this embodiment not only improves the accuracy and robustness of document parsing, but also ensures the interpretability and traceability of the results, making it particularly suitable for legal document analysis that requires high accuracy and transparency.

[0150] The aforementioned contract document parsing method based on multimodal adaptive feature fusion achieves efficient extraction and fusion of multimodal information by acquiring and preprocessing image and text data of contract documents. First, advanced encoding techniques are used to extract features from both images and text, and the semantic relevance between them is calculated to dynamically adjust weights for adaptive fusion, ensuring that different information sources are appropriately represented according to their importance during the parsing process. Next, the position and logical consistency of bounding boxes are optimized by capturing the contextual relevance of the contract text, significantly improving the accuracy of entity recognition and classification. Finally, the prediction results are bound to the original features to form a structured parsing report, providing detailed decision-making basis and explanation chains. This not only improves the overall accuracy of contract document parsing based on multimodal adaptive feature fusion but also enhances the transparency and traceability of the entire process, effectively addressing the shortcomings of existing technologies in multimodal feature fusion, bounding box generation accuracy, and parsing process transparency.

[0151] Figure 7 This is a schematic block diagram of a contract document parsing system 300 based on multimodal adaptive feature fusion provided in an embodiment of the present invention. Figure 7 As shown, corresponding to the above-described contract document parsing method based on multimodal adaptive feature fusion, this invention also provides a contract document parsing system 300 based on multimodal adaptive feature fusion. This contract document parsing system 300 includes a unit for executing the above-described contract document parsing method based on multimodal adaptive feature fusion, and the system can be configured in a server. Specifically, please refer to... Figure 7 The contract document parsing system 300 based on multimodal adaptive feature fusion includes an acquisition unit 301, a feature extraction unit 302, a fusion unit 303, an optimization unit 304, and a binding output unit 305.

[0152] The acquisition unit 301 is used to acquire image and text data of the contract document and perform preprocessing to obtain preprocessed image and contract text; the feature extraction unit 302 is used to extract multimodal features from the preprocessed image and contract text to obtain image features and text features; the fusion unit 303 is used to calculate the semantic relevance of the image features and the text features and perform adaptive weight fusion to obtain fused features; the optimization unit 304 is used to capture the contextual relevance of the contract text and optimize the position and logical consistency of the initial bounding box according to the fused features to obtain the prediction result; the binding output unit 305 is used to bind the prediction result with the image features and text features and output a structured parsing report.

[0153] In one embodiment, the acquisition unit 301 is used to acquire the image and text data of the contract document, perform resolution normalization and tilt correction on the image, perform OCR recognition and cleaning on the text data to remove noise characters, and obtain a structured text sequence to obtain the preprocessed image and contract text.

[0154] In one embodiment, the feature extraction unit 302 includes:

[0155] The text feature extraction subunit is used to perform semantic encoding on the preprocessed contract text using a fine-tuned BERT model to generate text features; the image feature extraction subunit is used to extract multi-scale visual features from the preprocessed image using an FPN network to obtain image features, wherein the image features include font style, layout structure, and key information such as table borders.

[0156] In one embodiment, the fusion unit 303 includes:

[0157] The similarity determination subunit is used to calculate the interaction features between the image features and the text features through a multi-head attention mechanism, and to determine the similarity between the image features and the text features; the complexity calculation subunit is used to calculate the text complexity and the image complexity respectively; the dynamic weight calculation subunit is used to calculate the dynamic weight based on the text complexity, the image complexity, and the similarity; and the feature fusion subunit is used to combine the dynamic weight to fuse the image features and the text features into a fused feature.

[0158] In one embodiment, the complexity calculation subunit includes:

[0159] The first calculation module is used to evaluate the image complexity by calculating a weighted sum of sentence length, legal term density, and syntax tree depth. The legal term density is based on the matching results of a preset legal term database, and the syntax tree depth is based on the number of nesting layers identified by the grammar rules. The second calculation module is used to obtain the image complexity by a weighted combination of edge detection score, color channel variance, and the number of object detections.

[0160] In one embodiment, the optimization unit 304 includes:

[0161] The context feature determination subunit is used to construct a context encoder using a bidirectional LSTM to capture semantic association features between clauses based on each text feature to obtain context features; the prediction subunit is used to generate candidate bounding boxes based on the layout detection model, and use the context features to perform semantic filtering and position fine-tuning on the candidate bounding boxes to obtain prediction results.

[0162] In one embodiment, the binding output unit 305 is used to bind the prediction result with the image features and text features to form a traceable parsing result, providing a detailed parsing decision path, including key thresholds, triggering keywords and visual features for merging / splitting operations, to obtain a structured parsing report and output the structured parsing report.

[0163] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the contract document parsing system 300 based on multimodal adaptive feature fusion and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0164] The aforementioned contract document parsing system 300 based on multimodal adaptive feature fusion can be implemented as a computer program, which can, for example... Figure 8 It runs on the computer device shown.

[0165] Please see Figure 8 , Figure 8 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0166] See Figure 8 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0167] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a contract document parsing method based on multimodal adaptive feature fusion.

[0168] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0169] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a contract document parsing method based on multimodal adaptive feature fusion.

[0170] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0171] The processor 502 is used to run the computer program 5032 stored in the memory to implement all the steps of the contract document parsing method based on multimodal adaptive feature fusion.

[0172] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0173] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0174] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein when executed by a processor, the computer program causes the processor to perform all the steps of the contract document parsing method based on multimodal adaptive feature fusion.

[0175] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0176] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0177] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of each unit is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0178] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the system of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0179] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0180] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A contract document parsing method based on multimodal adaptive feature fusion, characterized in that, include: The system acquires the image and text data of the contract document and performs preprocessing to obtain the preprocessed image and contract text. Multimodal feature extraction is performed on the preprocessed image and contract text to obtain image features and text features; The semantic correlation between the image features and the text features is calculated, and adaptive weight fusion is performed to obtain fused features; The contextual relevance of the contract text is captured, and the position and logical consistency of the initial bounding box are optimized based on the fusion features to obtain the prediction result; The prediction results are bound to the image features and text features to output a structured analysis report; The step of calculating the semantic relevance of the image features and the text features, and performing adaptive weight fusion to obtain fused features, includes: The interaction features between the image features and the text features are calculated using a multi-head attention mechanism to determine the similarity between the image features and the text features; Calculate the text complexity and image complexity separately; Calculate dynamic weights based on the text complexity, the image complexity, and the similarity. By combining dynamic weights, the image features and the text features are fused into a fused feature.

2. The contract document parsing method based on multimodal adaptive feature fusion according to claim 1, characterized in that, The process of acquiring the image and text data of the contract document and preprocessing it to obtain the preprocessed image and contract text includes: The process involves acquiring image and text data of the contract document, performing resolution normalization and tilt correction on the image, and performing OCR recognition and cleaning on the text data to remove noisy characters, thereby obtaining a structured text sequence and a preprocessed image and contract text.

3. The contract document parsing method based on multimodal adaptive feature fusion according to claim 1, characterized in that, The step of extracting multimodal features from the preprocessed image and contract text to obtain image features and text features includes: A fine-tuned BERT model is used to semantically encode the preprocessed contract text to generate text features. Multi-scale visual features are extracted from the preprocessed image using an FPN network to obtain image features, which include font style, layout structure, and key information such as table borders.

4. The contract document parsing method based on multimodal adaptive feature fusion according to claim 1, characterized in that, The calculation of text complexity and image complexity respectively includes: The evaluation is performed by calculating a weighted sum of sentence length, legal term density, and syntax tree depth. The legal term density is based on the matching results of a preset legal term database, while the syntax tree depth is based on the number of nesting levels identified by the grammar rules. Image complexity is obtained by weighting the edge detection score, color channel variance, and the number of object detections.

5. The contract document parsing method based on multimodal adaptive feature fusion according to claim 1, characterized in that, The process of capturing the contextual relevance of the contract text and optimizing the position and logical consistency of the initial bounding box based on fused features to obtain the prediction result includes: A bidirectional LSTM is used to construct a context encoder, which captures the semantic association features between clauses based on each text feature to obtain context features; Candidate bounding boxes are generated based on the layout detection model, and the context features are used to perform semantic filtering and position fine-tuning on the candidate bounding boxes to obtain the prediction results.

6. The contract document parsing method based on multimodal adaptive feature fusion according to claim 1, characterized in that, The step of binding the prediction results with the image features and text features, and outputting a structured parsing report, includes: The prediction results are bound to the image features and text features to form a traceable analysis result, providing a detailed analysis decision path, including key thresholds, triggering keywords and visual features for merging / splitting operations, to obtain a structured analysis report, and the structured analysis report is output.

7. A contract document parsing device based on multimodal adaptive feature fusion, characterized in that, The contract document parsing method based on multimodal adaptive feature fusion as described in any one of claims 1 to 6 includes: The acquisition unit is used to acquire the image and text data of the contract document and perform preprocessing to obtain the preprocessed image and contract text. The feature extraction unit is used to extract multimodal features from the preprocessed image and contract text to obtain image features and text features; The fusion unit is used to calculate the semantic correlation between the image features and the text features, and to perform adaptive weight fusion to obtain fused features; An optimization unit is used to capture the contextual relevance of the contract text and optimize the position and logical consistency of the initial bounding box based on the fusion features to obtain the prediction result; The binding output unit is used to bind the prediction result with the image features and text features, and output a structured parsing report.

8. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 6.

Images