A contract review method and system based on a multi-modal large model

Abstract

The application relates to the technical field of artificial intelligence and legal science, and discloses a contract examination method and system based on a multi-modal large model, which comprises the following steps: obtaining an image of an original contract document and encoding the image into a unified multi-modal feature representation that fuses text semantics, visual elements and their spatial coordinates; based on the structured rules in a contract examination knowledge base, extracting a feature subset related to the rules from the unified multi-modal feature representation and performing cross-modal reasoning; generating potential rule violation items and their supporting multi-modal evidence segments; for each potential rule violation item, checking the reliability of the evidence segments and generating a final examination result. The application realizes end-to-end joint modeling and understanding of contract text, visual elements and their spatial layout, can automatically, accurately and interpretively identify form and content violations of a contract, and significantly improves the accuracy, automation level and operability of contract examination.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and legal technology, and in particular to a contract review method and system based on a multimodal large model. Background Technology

[0002] Existing contract review systems primarily focus on textual modality processing, neglecting other modal information commonly found in contract documents. In actual business operations, contract documents often contain visual elements such as seals, signatures, tables, charts, and watermarks. These elements are crucial to the validity and completeness of the contract, and relying solely on textual analysis cannot fully assess the authenticity and compliance of the contract.

[0003] Some existing systems can process image information, but only in a limited way, processing each modality of data independently. They lack a deep understanding of the semantic relationship between text content and visual elements. For example, they cannot determine whether the seal position is correct, whether the signature matches the signatory's identity, or whether the table data is consistent with the text description.

[0004] Existing systems perform poorly when processing unstructured contract documents such as scanned or photographed documents. These documents may have problems such as skewing, blurriness, and background noise. The traditional serial processing method of OCR plus text analysis is prone to information loss and error accumulation.

[0005] Existing technologies often break down contract review into multiple independent sub-tasks, lacking the ability to comprehensively evaluate aspects such as the overall layout, formatting, and visual consistency of contract documents. Summary of the Invention

[0006] The purpose of this invention is to propose a contract review method and system based on a multimodal large model in order to solve the problems in the prior art.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a contract review method and system based on a multimodal large model, comprising the following steps:

[0008] S1. Obtain an image of the original contract document and input it into a pre-trained multimodal large model encoder. Output a unified multimodal feature representation that integrates text semantic features, visual element features and spatial coordinate features. The spatial coordinate features are used to characterize the coordinate position and layout relationship between visual elements and text content in the image.

[0009] S2. The unified multimodal feature representation is input into a dynamic rule-guided cross-modal reasoning module. This module encodes the triggering conditions of each structured rule into a rule query vector containing semantic query sub-vectors, visual query sub-vectors, and spatial query sub-vectors according to the structured rules in the preset contract review knowledge base. The module extracts the feature subsets related to the rules from the unified multimodal feature representation through the query vectors and performs cross-modal reasoning to generate an intermediate result containing potential violations and supporting multimodal evidence fragments.

[0010] S3. For each potential violation in the intermediate results, verify the reliability of its supporting multimodal evidence fragments, and generate a final review result based on the verification results. The final review result includes at least the determined violation type, the location information of the violation content in the original contract document image, and a natural language interpretation generated based on the evidence fragments.

[0011] A multimodal large model-based contract review system for implementing a multimodal large model-based contract review method, the system comprising:

[0012] The multimodal feature encoding module is used to acquire images of the original contract documents and process them using a pre-trained multimodal large model encoder to output a unified multimodal feature representation that integrates textual semantic features, visual element features, and spatial coordinate features.

[0013] The dynamic rule reasoning module, connected to the multimodal feature encoding module, is used to receive the unified multimodal feature representation and perform dynamic rule-guided cross-modal reasoning according to the structured rules in the preset contract review knowledge base to generate intermediate results containing potential violations and supporting multimodal evidence fragments.

[0014] The result verification and generation module is connected to the dynamic rule reasoning module. It is used to receive the intermediate results, verify the reliability of the supporting multimodal evidence fragments for each potential violation, and generate the final review result based on the verification results.

[0015] A contract review knowledge base, connected to the dynamic rule reasoning module, is used to store and provide the structured rules, each rule containing triggering conditions, review logic, and violation type;

[0016] The report output module, connected to the result verification and generation module, is used to receive the final review result and output it as a structured report containing the violation type, location information, and natural language interpretation.

[0017] The beneficial effects of the technical solution provided by this invention include at least the following:

[0018] This invention constructs an end-to-end multimodal unified modeling and understanding framework. It utilizes a pre-trained multimodal large model encoder to encode contract document images in an integrated manner, outputting a unified representation that integrates textual semantic features, visual element features, and spatial coordinate features. This method abandons the traditional serial process of OCR plus text analysis, realizes the integrated understanding of contract content and form, and significantly enhances the processing capability and robustness of contracts containing rich visual elements.

[0019] This invention innovatively proposes a dynamic rule-guided cross-modal reasoning mechanism. By encoding structured review rules into query vectors containing semantic, visual, and spatial sub-vectors, and dynamically matching and reasoning with the unified multimodal features of the document, a review result containing specific violation types and supporting multimodal evidence fragments is generated. This mechanism effectively overcomes the black-box decision-making problem that may exist in traditional large language models.

[0020] The final review report generated by the system of this invention not only points out the violations, but also visually highlights the violation areas through technologies such as salience weight distribution maps, and automatically generates evidence-based natural language explanations, which facilitates review by reviewers, reduces human-computer interaction costs, and provides auditors with a reliable intelligent auxiliary tool. Attached Figure Description

[0021] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart of the method provided in an embodiment of the present invention;

[0023] Figure 2 This is a system structure diagram provided for an embodiment of the present invention. Detailed Implementation

[0024] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a contract review method and system based on a multimodal large model proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0026] The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0027] The following description, in conjunction with the accompanying drawings, details a specific scheme for a contract review method and system based on a multimodal large model provided by this invention.

[0028] Please see Figure 1 The diagram illustrates a flowchart of a contract review method based on a multimodal large model according to an embodiment of the present invention. The method includes the following steps:

[0029] S1. Obtain an image of the original contract document and input it into a pre-trained multimodal large model encoder. Output a unified multimodal feature representation that integrates text semantic features, visual element features and spatial coordinate features. The spatial coordinate features are used to characterize the coordinate position and layout relationship between visual elements and text content in the image.

[0030] S2. The unified multimodal feature representation is input into a dynamic rule-guided cross-modal reasoning module. This module encodes the triggering conditions of each structured rule into a rule query vector containing semantic query sub-vectors, visual query sub-vectors, and spatial query sub-vectors according to the structured rules in the preset contract review knowledge base. The module extracts the feature subsets related to the rules from the unified multimodal feature representation through the query vectors and performs cross-modal reasoning to generate an intermediate result containing potential violations and supporting multimodal evidence fragments.

[0031] S3. For each potential violation in the intermediate results, verify the reliability of its supporting multimodal evidence fragments, and generate a final review result based on the verification results. The final review result includes at least the determined violation type, the location information of the violation content in the original contract document image, and a natural language interpretation generated based on the evidence fragments.

[0032] In one embodiment of the present invention, the pre-trained multimodal large model encoder is configured to perform the following operations:

[0033] The text content in the image of the original contract document is identified by optical character recognition, and the text semantic features of the text content are extracted by a text encoding subnetwork.

[0034] Visual elements that are not textual content in the image of the original contract document are extracted using a visual encoding subnetwork and used as visual element features.

[0035] The spatial location coding subnetwork extracts the coordinate positions and layout relationships between visual elements and text content in the original contract document image as spatial coordinate features;

[0036] By deeply fusing and aligning textual semantic features, visual element features, and spatial coordinate features, a unified multimodal feature representation is generated.

[0037] It should be noted that optical character recognition, text encoding subnetworks, visual encoding subnetworks, and spatial location encoding subnetworks can all be implemented using techniques commonly used in this field, including:

[0038] Optical character recognition: It can be implemented using any mature OCR engine or algorithm, such as CRNN, Attention-OCR and other text recognition models, or commercial OCR software interfaces. Its function is to accurately convert the text regions in the contract document image into a computer-readable text sequence.

[0039] Text encoding subnetwork: can be implemented using any mature pre-trained text encoding model, such as BERT, RoBERTa or ERNIE based on the Transformer architecture, and their variants based on recurrent neural networks. Its function is to map the input text sequence into text semantic features containing semantic information.

[0040] Visual encoding sub-network: It can be implemented using a general visual feature extraction model, such as ResNet, VGG or visual Transformer model. Its function is to extract visual element features from non-text areas such as stamps, signatures or table lines from images.

[0041] Spatial location encoding sub-network: It can be implemented using general object detection and location encoding techniques. For example, it can locate text blocks and visual elements through object detection models such as YOLO and Faster R-CNN, and obtain their bounding box coordinates. Then, through location embedding or geometric relationship graph construction methods, the bounding box coordinates and relative layout relationships are encoded into spatial coordinate features.

[0042] In one embodiment of the present invention, the step of deeply fusing and aligning text semantic features, visual element features, and spatial coordinate features to generate a unified multimodal feature representation includes:

[0043] Visual element features, text semantic features, and spatial coordinate features are mapped to the same high-dimensional latent space to obtain visual latent features, text latent features, and spatial latent features.

[0044] Based on visual latent features and textual latent features, the semantic association strength between each visual element and the text content in the image of the original contract document is analyzed and determined.

[0045] Based on spatial latent features, the relative positions and layout relationships of the visual elements they represent and the text in the image of the original contract document are determined, and the semantic association strength is corrected accordingly.

[0046] Based on the corrected semantic association strength, the visual latent features and textual latent features are weighted and fused to generate a cross-modal joint feature representation. The cross-modal joint feature representation is then normalized and nonlinearly transformed to generate the final unified multimodal feature representation.

[0047] It should be noted that, in this embodiment, the high-dimensional latent space refers to an abstract feature space of a unified dimension. It maps visual element features, text semantic features, and spatial coordinate features to a unified coordinate system that is comparable and operable through mathematical transformation. This mathematical transformation mapping process can be completed by the following formula:

[0048] Latent features = f(weight matrix·original features + bias vector);

[0049] This formula indicates that by combining a nonlinear activation function f with a linear transformation, the original features of different structures are transformed into visual latent features, textual latent features, and spatial latent features at the same semantic scale.

[0050] The core of determining the semantic association strength is to calculate the interaction relationship between visual latent features and textual latent features in the shared latent space. The specific process is as follows: using a set of textual latent features as queries and visual latent features as keys and values, calculate the dot product similarity between each textual latent feature and all visual latent features. After normalization by the Softmax function, a set of attention weights is obtained, which serve as the quantitative value of the semantic association strength. For example, the features describing the signer's text and the signature area will generate higher attention weights, while the weight between ordinary text and decorative icons will be close to zero.

[0051] Spatial latent features encode the absolute position and relative layout relationship between visual elements and text. The system inputs the spatial latent features into a multilayer perceptron and outputs a correction factor matrix. This correction factor matrix is ​​multiplied or added element-wise with the original semantic association strength to obtain a corrected semantic association strength that integrates both semantic and spatial evidence and conforms to the logic of real-world censorship. For example, the text "stamp here" is highly related to a certain stamp in latent spatial semantics, but the spatial coordinates of the stamp are far from the text coordinates, indicating that the spatial association between the two is weak. The correction factor will reduce its final association strength.

[0052] For each textual latent feature, the system performs a weighted summation of the corresponding visual latent features based on the strength of its modified semantic association with all visual elements, generating a visually relevant textual context vector. This textual context vector is concatenated with the original textual latent features and integrated into a unified cross-modal joint feature representation through a fully connected layer.

[0053] In one embodiment of the present invention, a unified multimodal feature representation is input into a dynamically rule-guided cross-modal reasoning module. This module extracts rule-related feature subsets from the unified multimodal feature representation based on structured rules in a pre-defined contract review knowledge base. It encodes the triggering conditions of each structured rule into a rule query vector containing semantic query sub-vectors, visual query sub-vectors, and spatial query sub-vectors. The step of generating an intermediate result containing potential violations and supporting multimodal evidence fragments through cross-modal reasoning using these query vectors includes:

[0054] Load at least one structured rule from the contract review knowledge base, each rule containing triggering conditions, review logic, and violation type;

[0055] The triggering conditions of each structured rule are converted into corresponding query vectors through a preset rule encoder, including semantic query sub-vectors, visual query sub-vectors, and spatial query sub-vectors.

[0056] The unified multimodal feature representation is separated and mapped to semantic, visual, and spatial feature subspaces that match the query subvectors through the corresponding feature decoupling module to obtain projection vectors, including: document semantic projection vectors that match the dimension of semantic query subvectors, document visual projection vectors that match the dimension of visual query subvectors, and document spatial projection vectors that match the dimension of spatial query subvectors.

[0057] Based on the query vector and projection vector, the relevance score sub-items between the unified multimodal feature representation and each structured rule are calculated and weighted to obtain the comprehensive relevance score;

[0058] Rules with a comprehensive relevance score higher than a preset activation threshold are dynamically activated, and for each activated rule, a subset of features constituting evidence clues is located and selected from the unified multimodal feature representation.

[0059] For each activated rule, cross-modal reasoning is performed on its corresponding feature subset. Through joint analysis of textual, visual, and spatial features, it is determined whether the corresponding review logic is triggered. If triggered, a potential violation is generated, and the corresponding feature subset is converted into an interpretable supporting multimodal evidence fragment.

[0060] Aggregate all potential violations and their corresponding supporting multimodal evidence fragments to generate intermediate results.

[0061] It should be noted that the pre-built contract review knowledge base is a structured and scalable set of rules. Its essence is to transform expert knowledge, legal and regulatory requirements, and industry practices in the field of contract review into machine-readable logical rules. This knowledge base supports dynamic import and configuration, allowing users to load different subsets of rules according to different contract types, thereby achieving flexibility and targeting in the review.

[0062] The pre-defined rule encoder first receives a description of the triggering conditions of a structured rule, which includes explicit text keywords, visual object categories and spatial relationship phrases. The encoder contains three parallel sub-encoders: (1) Semantic sub-encoder: encodes the text part into a fixed-dimensional semantic query sub-vector to represent the text concept that the rule is concerned with; (2) Visual sub-encoder: encodes the description of visual elements in the rule into a visual query sub-vector through visual vocabulary embedding or object detection prototype vectors to represent the feature pattern of the expected visual object; (3) Spatial sub-encoder: usually a geometric relationship encoder, used to transform the spatial relationship description in the rule into a computable spatial query sub-vector.

[0063] The feature decoupling module corresponds to the rule encoder, decoupling the unified multimodal features and projecting them into three specific subspaces aligned with the rule query vectors. It includes three independent projection networks: a semantic projection network, a visual projection network, and a spatial projection network, so that each projection vector can represent the conformity of the original contract document with the rule conditions in the corresponding modality in its respective subspace.

[0064] In one embodiment of the present invention, the step of calculating and weighting the relevance score sub-items between the unified multimodal feature representation and each structured rule based on the query vector and projection vector to obtain a comprehensive relevance score includes:

[0065] Calculate the degree of matching between the query vector and the projection vector in their respective subspaces, including:

[0066] Calculate the semantic similarity between the semantic query subvector and the document semantic projection vector, and use it as the semantic relevance score;

[0067] Calculate the visual matching degree between the visual query subvector and the document visual projection vector, and use it as a visual relevance score;

[0068] The degree of conformity between the constraints defined by the spatial query subvector and the actual layout described by the document spatial projection vector is evaluated as a spatial relevance score.

[0069] The semantic relevance score, visual relevance score, and spatial relevance score are weighted and fused to obtain the comprehensive relevance score.

[0070] It should be noted that the following are specific examples of calculating semantic relevance score, visual relevance score, and spatial relevance score:

[0071] Semantic relevance score:

[0072]

[0073] In the formula, The semantic relevance score. For semantic query sub-vectors, For document semantic projection vectors;

[0074] Visual relevance score:

[0075]

[0076] In the formula, Visual relevance score, For visual query subvectors, For document visual projection vectors, and They are respectively The sensitivity parameters and bias terms of the function;

[0077] Spatial correlation score:

[0078]

[0079] In the formula, Spatial correlation score, For spatial query subvectors, For document space projection vectors, For standard exponential functions, for Sensitivity parameters, For distance measurement function, For a purpose of Instantiated as a decoder for mathematical constraints.

[0080] The three relevance scores above are weighted and fused, with the weights automatically calculated by a lightweight dynamic weight generation network, including:

[0081] The trigger condition query vector and its corresponding unified multimodal feature representation are concatenated to generate a joint context feature C. This joint context feature C is then input into a multilayer perceptron (MPL) with three output nodes. The outputs of the three nodes are normalized using the Softmax function to obtain the weights corresponding to the three relevance scores. , and Used for weighted summation:

[0082]

[0083] In the formula, To achieve a comprehensive relevance score, The weights for semantic relevance scores, The weights for visual relevance scores, The weights for spatial relevance scores.

[0084] In one embodiment of the present invention, the step of dynamically activating rules whose comprehensive relevance scores are higher than a preset activation threshold, and for each activated rule, locating and filtering a subset of features constituting evidence clues from a unified multimodal feature representation, includes:

[0085] Based on the comparison between the comprehensive relevance score and the preset activation threshold, the rules with a comprehensive relevance score that is higher than or equal to the preset activation threshold are identified as the set of rules to be activated.

[0086] For each rule in the set of rules to be activated, obtain the intermediate association data generated in the process of calculating the comprehensive relevance score of the rule, including relevance score sub-items, query vectors and projection vectors used to calculate relevance score sub-items, and weights used for weighted fusion of relevance score sub-items;

[0087] A significance weight distribution map is generated based on intermediate correlation data;

[0088] Based on the saliency weight distribution map, feature units with weights higher than or equal to the preset focusing threshold are extracted from the unified multimodal feature representation to form a feature subset.

[0089] Furthermore, the step of generating a significance weight distribution map based on intermediate association data includes:

[0090] Based on the semantic query vector and document semantic projection vector obtained in the process of calculating semantic relevance score, the reverse attention mapping method is used to align them with the text content of the original contract document to generate a preliminary semantic saliency map.

[0091] Based on the visual query vector and document visual projection vector obtained during the calculation of visual relevance scores, the visual query vector and document visual projection vector are aligned with the visual elements of the original contract document using the feature map space weight backtracking method to generate an initial visual saliency map.

[0092] Based on the spatial query vector and document spatial projection vector obtained during the calculation of spatial relevance score, a spatial conformity distribution map is generated.

[0093] Based on the weights used for weighted fusion of relevance score sub-items, the initial semantic saliency map, the initial visual saliency map, and the spatial conformity distribution map are weighted and fused to generate a saliency weight distribution map.

[0094] It should be noted that the specific process of generating the initial semantic saliency map includes: using the semantic query vector generated during the calculation of the semantic relevance score as the key and the document semantic projection vector as the query, calculating the cross-attention weight of each key and query to obtain the cross-attention weight matrix, and upsampling this matrix along the text sequence dimension of the original contract document to the same resolution as the original contract document image through bilinear interpolation or transpose convolution to generate the initial semantic saliency map.

[0095] The specific process of generating the initial visual saliency map includes: calculating the cosine similarity between the visual query vector and the document visual projection vector in the channel dimension to obtain the channel attention weights; further, weighting the channel attention weights with the feature map formed by all visual projection vectors; and aligning the weighted feature map with the original contract document image through class activation mapping technology to generate the initial visual saliency map.

[0096] The specific process of generating the spatial compliance distribution map includes: parsing the spatial query vector into a geometric constraint function, decoding the document spatial projection vector into a set of actual geometric parameters, calculating the probability value of each coordinate position of the original contract document image satisfying the constraint function, and forming the spatial compliance distribution map.

[0097] In the saliency weight distribution map, each weighted pixel or grid point explicitly to one or more feature vectors in the unified multimodal feature representation, which originate from a specific local region and / or a specific semantic segment of the original contract document.

[0098] The preset activation threshold is designed to balance review coverage and system efficiency. In practice, a fixed value based on empirical statistics (e.g., 0.55) can be used, or it can be dynamically configured according to the risk level or business priority of the rule. For example, for high-risk rules involving core amounts or legal responsibilities, a lower activation threshold (e.g., 0.4) can be used to ensure that sensitive rules are not missed, while for general format check rules, a higher threshold (e.g., 0.6) can be used to improve efficiency.

[0099] The purpose of setting a preset focus threshold is to accurately locate key evidence areas from the saliency weight distribution map. In practice, a dynamic threshold setting method based on distribution statistics can be adopted. That is, for each saliency weight distribution map, the statistical characteristics of its weights are calculated, usually the mean μ and the standard deviation σ. The preset focus threshold is set to μ + k × σ, where k is an adjustable parameter (e.g., k = 1.0). This method can adapt to different contract documents to be reviewed and the differences in weight distribution generated by different rules.

[0100] The preset activation threshold and focus threshold, along with their calculation rules, can be stored in a preset contract review knowledge base for easy access at any time.

[0101] In one embodiment of the present invention, for each potential violation in the intermediate results, the reliability of its supporting multimodal evidence fragments is verified, and a final review result is generated based on the verification results. The final review result includes at least the determined violation type, the location information of the violation content in the original contract document image, and the natural language interpretation generated based on the evidence fragments.

[0102] Analyze the intermediate results to obtain the potential violations contained therein and their associated supporting multimodal evidence fragments;

[0103] For each potential violation, a confidence score is calculated based on the clarity, completeness, and correlation strength of its supporting multimodal evidence fragments with the corresponding violation type.

[0104] The confidence score is compared with a preset confidence threshold, and the following decision is made:

[0105] If a piece of evidence passes the consistency check and its confidence score is higher than or equal to the confidence threshold, then the potential violation is marked as a confirmed violation.

[0106] If a piece of evidence fails the consistency check or the confidence score is lower than the confidence threshold, the potential violation will be rejected or marked as a potential violation pending review.

[0107] For each confirmed violation, a structured final review result is generated. All generated final review results are then aggregated to form a complete contract review report.

[0108] It should be noted that the calculation of the confidence score includes: (1) calculating the clarity score by calculating the L2 norm of each evidence vector in the multimodal evidence fragment; (2) calculating the integrity score by the ratio of the actual number of extracted elements to the total number of elements required by the rule; (3) reusing the comprehensive relevance score corresponding to the rule as the association strength score of the violation type; (4) finally normalizing the three to a confidence score in the range of [0, 1].

[0109] The confidence threshold can be obtained by analyzing the confidence score distribution of "confirmed violations" and "rejected violations" on a system trained and validated with a large number of contract samples.

[0110] As one embodiment of the present invention, the structured final review results include at least:

[0111] The confirmed violation type, i.e. the violation category corresponding to the triggering rule;

[0112] Location information refers to the coordinate areas or anchor points in the original contract document image used to identify the location of the violation content;

[0113] Natural language interpretation refers to explanatory text automatically generated based on verified evidence fragments, used to describe the facts and basis of violations.

[0114] Please see Figure 2 The diagram illustrates a system architecture of a contract review system based on a multimodal large model, according to an embodiment of the present invention. The system includes:

[0115] The multimodal feature encoding module is used to acquire images of the original contract documents and process them using a pre-trained multimodal large model encoder to output a unified multimodal feature representation that integrates textual semantic features, visual element features, and spatial coordinate features.

[0116] The dynamic rule reasoning module, connected to the multimodal feature encoding module, is used to receive unified multimodal feature representations and perform dynamic rule-guided cross-modal reasoning based on the structured rules in the preset contract review knowledge base to generate intermediate results containing potential violations and supporting multimodal evidence fragments.

[0117] The result verification and generation module, connected to the dynamic rule reasoning module, is used to receive intermediate results, verify the reliability of the supporting multimodal evidence fragments for each potential violation, and generate the final review result based on the verification results.

[0118] The contract review knowledge base, connected to the dynamic rule reasoning module, is used to store and provide structured rules. Each rule includes triggering conditions, review logic, and violation type.

[0119] The report output module, connected to the result verification and generation module, is used to receive the final review results and output them as a structured report containing violation type, location information, and natural language interpretation.

[0120] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A contract review method based on a multi-modal large model, characterized in that, The method comprises: S1, obtaining the image of the original contract document and inputting it into a pre-trained multi-modal model encoder to output a unified multi-modal feature representation fusing text semantic features, visual element features and their spatial coordinate features, wherein the spatial coordinate features are used to represent the coordinate position and layout relationship of the visual elements and the text content in the image; S2, inputting the unified multi-modal feature representation into a dynamic rule guided cross-modal reasoning module, which encodes the trigger condition of each structured rule in the contract review knowledge base into a rule query vector containing a semantic query sub-vector, a visual query sub-vector and a spatial query sub-vector according to the structured rules in the contract review knowledge base, extracts the feature subset related to the rule from the unified multi-modal feature representation through the query vector, and performs cross-modal reasoning to generate an intermediate result containing potential violation items and their supporting multi-modal evidence fragments; The step S2 comprises: loading at least one structured rule from the contract review knowledge base, wherein each rule contains a trigger condition, a review logic and a violation type; converting the trigger condition of each structured rule into a corresponding query vector through a pre-set rule encoder, including a semantic query sub-vector, a visual query sub-vector and a spatial query sub-vector; separating and mapping the unified multi-modal feature representation to the semantic, visual and spatial feature subspaces matched with the query sub-vectors through a corresponding feature decoupling module to obtain projection vectors, including a document semantic projection vector matched with the semantic query sub-vector dimension, a document visual projection vector matched with the visual query sub-vector dimension, and a document spatial projection vector matched with the spatial query sub-vector dimension; calculating the correlation score sub-items between the unified multi-modal feature representation and each structured rule based on the query vector and the projection vector and performing weighted fusion to obtain a comprehensive correlation score; dynamically activating the rules with a comprehensive correlation score higher than a pre-set activation threshold, and for each activated rule, locating and screening a feature subset constituting evidence clues from the unified multi-modal feature representation; performing cross-modal reasoning on the corresponding feature subset of each activated rule, and through joint analysis of text, visual and spatial features, determining whether the corresponding review logic is triggered, if triggered, generating a potential violation item, and converting the corresponding feature subset into an interpretable supporting multi-modal evidence fragment; aggregating all the potential violation items and their corresponding supporting multi-modal evidence fragments to generate the intermediate result; The step of dynamically activating the rules with a comprehensive correlation score higher than a pre-set activation threshold, and for each activated rule, locating and screening a feature subset constituting evidence clues from the unified multi-modal feature representation comprises: determining the rules with a comprehensive correlation score higher than or equal to the pre-set activation threshold as the rule set to be activated according to the comparison result of the comprehensive correlation score and the pre-set activation threshold. For each rule in the set of rules to be activated, obtain intermediate association data generated in the process of calculating the comprehensive correlation score of the rule, including the correlation score sub-items, the query vector and the projection vector used to calculate the correlation score sub-items, and the weight used for the weighted fusion of the correlation score sub-items; Based on the intermediate association data, generate a saliency weight distribution map; According to the saliency weight distribution map, extract feature units with a weight higher than or equal to a preset focus threshold from the unified multi-modal feature representation to form the feature subset; S3, for each potential violation item in the intermediate result, check the reliability of the supporting multi-modal evidence segment, and generate a final review result according to the check result, the final review result at least includes the determined violation type, the positioning information of the violation content in the original contract document image and the natural language explanation generated based on the evidence segment.

2. The multi-modal large model-based contract review method according to claim 1, characterized in that: The pre-trained multi-modal large model encoder is configured to perform the following operations: extracting text semantic features of the text content in the image of the original contract document through a text encoding sub-network by optical character recognition; extracting visual elements of non-text content in the image of the original contract document as visual element features through a visual encoding sub-network; extracting the coordinate position and layout relationship of the visual elements and the text content in the image of the original contract document as the spatial coordinate features through a spatial position encoding sub-network; deeply fusing and aligning the text semantic features, visual element features and spatial coordinate features to generate a unified multi-modal feature representation.

3. The multi-modal large model-based contract review method according to claim 2, characterized in that: The step of deeply fusing and aligning the text semantic features, visual element features and spatial coordinate features to generate a unified multi-modal feature representation includes: mapping the visual element features, text semantic features and spatial coordinate features to the same high-dimensional hidden space respectively to obtain visual hidden features, text hidden features and spatial hidden features; based on the visual hidden features and text hidden features, analyzing and determining the semantic association strength between each visual element and text content in the image of the original contract document; based on the spatial hidden features, determining the relative position and layout relationship of the visual elements and texts represented thereby in the image of the original contract document, and correcting the semantic association strength accordingly; based on the corrected semantic association strength, weightedly fusing the visual hidden features and text hidden features to generate a cross-modal joint feature representation, and further normalizing and nonlinearly transforming the cross-modal joint feature representation to generate a final unified multi-modal feature representation.

4. The multi-modal large model-based contract review method according to claim 1, characterized in that: The step of calculating and weighting the correlation score between the unified multi-modal feature representation and each structured rule based on the query vector and the projection vector includes: respectively calculating the matching degree of the query vector and the projection vector in the corresponding subspace, including: calculating the semantic similarity between the semantic query sub-vector and the document semantic projection vector as a semantic correlation score; calculating the visual matching degree between the visual query sub-vector and the document visual projection vector as a visual correlation score; evaluating the compliance between the constraint condition defined by the spatial query sub-vector and the actual layout described by the document spatial projection vector as a spatial correlation score; weighting and fusing the semantic correlation score, the visual correlation score and the spatial correlation score to obtain a comprehensive correlation score.

5. The multi-modal large model-based contract review method according to claim 1, characterized in that: The step of generating the saliency weight distribution map based on the intermediate association data includes: Based on the semantic query vector and the document semantic projection vector obtained in the process of calculating the semantic correlation score, a semantic saliency preliminary map is generated by aligning them with the text content of the original contract document through a reverse attention mapping method. Based on the visual query vector and the document visual projection vector obtained in the process of calculating the visual correlation score, a visual saliency preliminary map is generated by aligning them with the visual elements of the original contract document through a feature map space weight backtracking method. Based on the spatial query vector and the document spatial projection vector obtained in the process of calculating the spatial correlation score, a spatial compliance distribution map is generated. Based on the weights for weighting and fusing the correlation score sub-items, the semantic saliency preliminary map, the visual saliency preliminary map and the spatial compliance distribution map are weighted and fused to generate a saliency weight distribution map.

6. The multi-modal large model-based contract review method according to claim 1, characterized in that: The step of checking the reliability of the supporting multi-modal evidence segments of each potential violation item in the intermediate result and generating a final review result according to the checking result, which at least includes the determined violation type, the positioning information of the violation content in the original contract document image and the natural language explanation generated based on the evidence segments, includes: Parsing the intermediate result to obtain the potential violation items and the associated supporting multi-modal evidence segments contained therein; For each potential violation item, the confidence score of the potential violation item is calculated based on the clarity, integrity of the supporting multi-modal evidence segments and the association strength with the corresponding violation type; The confidence score is compared with the preset confidence threshold, and the following decisions are executed: If the evidence segments pass the consistency check and the confidence score is higher than or equal to the confidence threshold, the potential violation item is marked as a confirmed violation item; If the evidence segments do not pass the consistency check, or the confidence score is lower than the confidence threshold, the potential violation item is rejected or marked as a potential violation item to be reviewed. For each confirmed violation item, a structured final review result is generated, all generated final review results are aggregated to form a complete contract review report.

7. The contract review method based on a multi-modal large model according to claim 6, characterized in that: wherein the structured final review result at least includes: a confirmed violation type, i.e. a violation classification corresponding to a trigger rule; positioning information, i.e. a coordinate area or anchor point in the original contract document image for identifying the location of the violation content; a natural language explanation, i.e. an explanatory text automatically generated based on the verified evidence fragment, for describing the violation fact and the basis. 8.A multi-modal large model based contract review system for performing a multi-modal large model based contract review method, characterized by, The system comprises: a multi-modal feature encoding module for obtaining an image of an original contract document and processing it using a pre-trained multi-modal large model encoder to output a unified multi-modal feature representation that integrates text semantic features, visual element features and their spatial coordinate features; a dynamic rule reasoning module connected to the multi-modal feature encoding module for receiving the unified multi-modal feature representation and performing dynamic rule-guided cross-modal reasoning according to structured rules in a pre-set contract review knowledge base to generate an intermediate result containing potential violation items and their supporting multi-modal evidence fragments; a result verification and generation module connected to the dynamic rule reasoning module for receiving the intermediate result and performing reliability verification on the supporting multi-modal evidence fragments of each potential violation item, and generating a final review result according to the verification result; a contract review knowledge base connected to the dynamic rule reasoning module for storing and providing the structured rules, each rule containing a trigger condition, a review logic and a violation type; a report output module connected to the result verification and generation module for receiving the final review result and outputting it as a structured report containing the violation type, the positioning information and the natural language explanation.

Images