Intelligent contract content processing method based on multiple modes

Abstract

The invention provides a contract content intelligent processing method based on multi-modal collaboration, and relates to the technical field of document intelligent processing, and the method comprises the steps: firstly, carrying out the preprocessing of a contract image, and fusing the multi-modal information, such as a printed text, a table, a signature and handwritten content, into structural data with a semantic tag through OCR, Layout analysis and visual large model collaborative analysis; semantic partitioning is carried out according to contract hierarchies, a general semantic vector and a contract field special vector are generated for each semantic block, and an index database containing page numbers, coordinates and multi-dimensional vectors is constructed; when a user inputs an auditing index, positioning a related semantic block by adopting a keyword matching and multi-vector mixed retrieval strategy; and finally, performing visual large model closed-loop verification on the key auditing item directly based on the original image. According to the method, the special prompt word template library in the contract field is constructed by introducing the fe-shot learning, and the retrieval result is dynamically optimized by reordering the cross encoder, so that the accuracy and the reliability of contract auditing are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent document processing technology, and more specifically to a multimodal intelligent contract content processing method. Background Technology

[0002] In finance, legal affairs, and corporate operations, contracts, as crucial legal documents, are paramount in terms of accuracy, completeness, and compliance. Traditional contract review relies on manual reading and judgment by legal personnel, which is inefficient and prone to overlooking risks due to fatigue or negligence. With the accelerating pace of enterprise digitalization, the need for efficient processing of massive volumes of contracts is increasingly urgent, making intelligent contract processing systems a hot topic in industry research and development.

[0003] Currently, mainstream contract processing methods typically rely on OCR technology to convert paper or image-format contracts into text, followed by Natural Language Processing (NLP) models for keyword matching, entity recognition, or semantic analysis. However, these methods suffer from significantly higher error rates in real-world contracts with complex and diverse content, such as handwritten signatures, annotations, low-resolution scans, stamped text, and special or artistic fonts. This introduces inherent noise into the input source for subsequent semantic analysis. Traditional OCR often outputs linear, unstructured text streams, failing to effectively recreate the original contract layout. To address these issues, multimodal visual language models such as LLaVA, Qwen-VL, and GPT-4V can directly understand text, layout, handwritten content, and visual elements within images, generating high-quality structured descriptions.

[0004] Therefore, this invention provides a multimodal intelligent processing method for contract content, which comprehensively utilizes multimodal information to break through the technical bottleneck of traditional OCR, realizes the transformation from text dependence to visual understanding. The multimodal collaborative solution can improve the accuracy and reliability of automated contract processing, while taking into account processing efficiency and result accuracy, and achieve full-element, highly robust intelligent processing of contract content. Summary of the Invention

[0005] To address the shortcomings of the existing technologies, the technical problem to be solved by this invention is to provide an intelligent contract content processing method based on multimodal collaboration, which solves the problem of low recognition accuracy for complex and diverse contract documents, realizes high-fidelity restoration and intelligent review of multimodal contract information, and improves processing efficiency and review accuracy.

[0006] The technical solution adopted in this invention is to provide a multimodal intelligent processing method for contract content. First, the contract image is preprocessed. Then, through OCR, layout analysis, and collaborative parsing with a large visual model, multimodal information such as printed text, tables, signatures, and handwritten content is fused into structured data with semantic tags. Semantic blocks are formed according to the contract hierarchy, and a general semantic vector and a contract-domain-specific vector are generated for each semantic block. An index database containing page numbers, coordinates, and multidimensional vectors is constructed. When the user inputs review indicators, a keyword matching and multi-vector hybrid retrieval strategy is used to locate relevant semantic blocks. Finally, key review items are directly verified by a large visual model based on the original image. Accurate parsing results are obtained by optimizing prompt words, ensuring the accuracy and reliability of the review.

[0007] In a preferred embodiment, the present invention provides a multimodal collaborative parsing architecture, which integrates OCR, layout analysis, and a large visual model to achieve unified parsing of printed text, tables, seals, handwritten content, etc. It also utilizes the large visual model to directly generate structured descriptions, replacing the traditional OCR for recognizing non-standard content, significantly improving the semantic understanding of elements such as tables, handwriting, and seals. Furthermore, it introduces an automatic prompt word optimization mechanism, and builds a contract-specific prompt word template library based on few-shot learning, allowing the model to quickly understand and adapt to new tasks by observing a very small number of examples, thereby improving parsing accuracy.

[0008] In a preferred embodiment, the present invention provides a dynamic optimization method for retrieval strategies. This method combines keyword matching, general semantic vectors, and contract-specific vectors to implement retrieval strategies. The weights of each retrieval method are determined based on experience. A cross-transformer is introduced for re-ranking. Deep interactive modeling is used to accurately calculate the true relevance between user review indicators and candidate semantic blocks, enabling the model to master the judgment rules of the contract domain. Dynamically optimizing the ranking results allows the model to adapt to the ever-changing review needs in actual business. The lightweight model balances efficiency and accuracy. Attached Figure Description

[0009] The present invention will be further described below with reference to the accompanying drawings and embodiments: Figure 1 This is a structural diagram of a multimodal intelligent contract content processing method according to the present invention; Figure 2 This is a flowchart of a multimodal collaborative parsing architecture according to the present invention; Figure 3 This is a flowchart of a dynamic optimization method for a retrieval strategy according to the present invention. Detailed Implementation

[0010] To better understand the purpose, system architecture, and functional implementation of this embodiment, the embodiments and features described herein can be combined with each other without conflict. The exemplary embodiments disclosed herein will be described below with reference to the accompanying drawings, including specific technical details disclosed to aid understanding; however, these details should be considered exemplary rather than restrictive. Therefore, those skilled in the art should understand that various improvements and adjustments can be made to the embodiments described herein without departing from the scope and core ideas of the invention. Similarly, for clarity, detailed descriptions of well-known technologies, functions, and structures are omitted in the following description.

[0011] Example 1 Figure 1 This is a structural diagram of a multimodal intelligent contract content processing method according to the present invention.

[0012] like Figure 1 As shown, the implementation steps of a multimodal intelligent contract content processing method 100 include S110~S140.

[0013] In step S110, multimodal collaborative parsing, combined with OCR, layout analysis, and visual large model, generates structured data.

[0014] In step S120, the index construction, semantic segmentation, and domain bi-vector index construction are merged to establish the database.

[0015] In step S130, intelligent retrieval optimization is performed, keyword and dual-vector retrieval are combined, and the cross encoder reorders to adapt to business changes.

[0016] In step S140, closed-loop verification is performed, key items are determined to determine whether visual verification is required, and Few-shot prompts are optimized.

[0017] According to an embodiment of the present invention, in step S110, OCR is used to efficiently extract conventional printed text, ensuring the efficiency of basic text acquisition. In layout analysis, the Transformer model based on the PubLayNet dataset is used to accurately identify the title level, paragraph division, table or seal position of the contract, and output the spatial coordinates of the identified position. The visual big model directly generates structured descriptions for unconventional printed text such as handwritten signatures, complex tables and text within seals, unifying all content into structured data with semantic tags, and retaining page numbers and coordinate metadata.

[0018] According to an embodiment of the present invention, in step S120, a multidimensional index is constructed by constructing text content, spatial location, and dual vector embedding. The text is segmented according to the contract's clause hierarchy to ensure the semantic integrity of each text block. Page number and image coordinate information are associated with each text block for subsequent accurate positioning of the original image. The dual vector embedding includes two types of vectors: general semantic vectors and contract domain-specific vectors. The semantic matching of contract terminology is strengthened based on the contract content corpus, avoiding the semantic bias of the single vector model and laying a solid foundation for accurate retrieval.

[0019] According to an embodiment of the present invention, in step S130, a triple retrieval fusion strategy is adopted. First, keyword precision matching is performed to lock highly relevant text blocks for explicit terms such as "signing date" and "monthly rent". Second, vector retrieval is performed using the BGE-M3 model to cover synonyms or similar expressions such as "payment event" and "payment date" to improve the recall rate of the model retrieval. Then, for industry-specific expressions, such as the legal differences between "deposit" and "down payment", the accuracy of professional semantic matching is enhanced. Finally, through comprehensive ranking, different weights are applied to keywords, general vectors and domain vectors to output the optimal result.

[0020] According to an embodiment of the present invention, in step S140, image coordinates are obtained by locating text blocks through hybrid retrieval to crop the original image region. Finally, the visual big model directly parses and outputs the results. In the review environment, it does not rely on intermediate OCR results. Based on the retrieved and located coordinates, the visual big model is directly called to parse information from the original image, thereby avoiding errors generated in OCR recognition.

[0021] Figure 2 This is a flowchart of a multimodal collaborative parsing architecture of the present invention.

[0022] like Figure 2 As shown, when a user initiates a contract review request, the system clarifies the core review items and elements for different contract types, defines the content boundaries for the prompt word template, and classifies contract types into categories such as leasing, procurement, labor services, and loans according to industry standards. Within each type, the system further subdivides the required scenarios. For each type of contract, the system extracts key review items and related elements through domain expert annotation or legal text analysis, and clarifies the review points that users are concerned about. This serves as the starting point for domain adaptation and provides a basis for subsequent use of domain-specific prompt words and industry-customized visual models.

[0023] A few-shot learning-based prompt word template library is constructed. First, a general prompt word template is generated, and a general structure is designed based on element parsing requirements. Then, domain-adapted fine-tuning is performed. For each type of contract, the general template is manually modified to adapt to domain characteristics, forming a small number of high-quality example templates as a support set for few-shot learning. The prompt word style and element priority of different contract types are clarified. For review items in the corresponding contract type that have not generated templates, only the review item name and element list are provided as the query set for model input. A meta-generative model of LLaMA combined with a prototype prompting mechanism is used for training and inference. The model learns the mapping rules between review items and template structures in the support set. During the inference phase, the query set is input, and the model generates corresponding domain templates based on the rules of the support set. Rule validation and manual sampling are used to filter invalid templates and retain high-quality templates for inclusion in the library. Finally, through few-shot learning, the model automatically generates prompt word templates for other review items of the same type of contract based on the support set, thereby expanding the template library.

[0024] Based on contract type and review indicators, historical successful examples are retrieved from the domain-specific prompt word template library. All historical examples in the example library are converted into contract domain-specific vectors to facilitate similarity calculation. At the same time, the user's current review requirements are also converted into contract domain vectors. The matching degree between the current question vector and the example library vector is calculated using the cosine similarity algorithm. The most suitable historical examples are selected to ensure that the most valuable examples are found. The calculation formula is shown in the following formula (1).

[0025] (1) in, Let this be the current problem vector. For example library vectors, For vector dot product, and It is the L2 norm of the vector, and its value range is... The closer the similarity is to 1, the more semantically similar the terms. Based on the most similar historical cases and the specific context of the current contract, domain-specific prompts are generated.

[0026] The final prompt words and the semantic blocks located in the contract image are input into the LLaVA (Large Language and Vision Assistant) visual model, which directly parses information from the image modality. LLaVA first extracts the visual features of the semantic blocks in the contract image through a visual encoder. As shown in equation (2) below.

[0027] (2) in, For Transformer encoders, For the features of the i-th image, local visual information is captured through visual encoding, such as the strokes of the text in the contract, the lines of the table, and the color of the seal.

[0028] LLaVA uses a learnable linear projection layer By mapping visual features to the input space of a language model to achieve cross-modal alignment, LLaVA adds a special image start marker feature before the projected visual features. Ultimately, the cross-modal mapping of visual prefix features into the input language model ensures that visual information can be understood by the language model. The LLaVA language decoder uses visual prefix features and cue word token features as input sequence features. The decoder generates a structured parsing result sequence based on the input sequence through autoregression. The generation probability of each token depends on all the previous tokens and input features, as shown in the following formula (3).

[0029] (3) in, Calculated for the softmax output layer of the language model. To parse the result sequence, the token sequence with the highest probability is finally selected as the parsing result. The visual big data model completes the closed loop of the entire process from user request to accurate review result by outputting structured data with semantic labels.

[0030] Figure 3 This is a flowchart of a dynamic optimization method for a retrieval strategy according to the present invention.

[0031] like Figure 3 As shown, when a user initiates a search request for contract review, a triple hybrid detection strategy of keyword matching, general vector retrieval, and domain vector retrieval is used. Keyword matching uses string matching to quickly locate highly relevant text blocks for professional terms. General vector retrieval transforms the user query into a general semantic vector and calculates the similarity between the transformed general semantic vector and the general vector in the text library of the contract, thereby covering synonyms and similar expressions. Domain vector retrieval calls the contract domain-specific vector to strengthen the semantic matching of professional terms. The results of the above three retrievals are merged and deduplicated to obtain a batch of candidate text blocks.

[0032] A cross-encoder is used to deeply model the semantic interaction between candidate text blocks and user queries, reordering the results. First, a hybrid keyword and multi-vector retrieval method is used to quickly recall candidate semantic blocks. Each recalled candidate semantic block is concatenated with the user query to form an input sequence, which is then input into the cross-encoder to obtain a relevance score. Finally, the candidate blocks are reordered from high to low scores to filter out low-quality matches, outputting the reordered contract text block as the core basis for user review. A dynamic optimization mechanism allows the model to adapt to the ever-changing review needs in actual business, continuously optimizing the ranking effect. A result adjustment function is added to the contract review interface, allowing users to manually adjust the ranking of search results and mark incorrect matches. User behavior data is automatically recorded as a weak supervision signal. User feedback data is periodically transformed into new training samples to update the cross-encoder, avoiding the high cost of full retraining while retaining historical learning results.

[0033] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0034] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for intelligent processing of contract content based on multimodal collaboration, characterized by: Includes the following steps: S1. Multimodal collaborative parsing: Through OCR, layout analysis and visual big model collaborative parsing, multimodal information such as printed text, tables, signatures and handwritten content are integrated into structured data with semantic tags. S2. Semantic segmentation and index construction: Semantic segmentation is performed according to the contract hierarchy, and a general semantic vector and a contract domain-specific vector are generated for each semantic block to build an index database containing page numbers, coordinates and multi-dimensional vectors. S3. Hybrid retrieval and positioning: When the user inputs the audit criteria, a hybrid retrieval strategy of keyword matching and multi-vector is used to locate the relevant semantic blocks. S4, Visual Large Model Closed-Loop Verification: For key review items, visual large model closed-loop verification is performed directly based on the original image, and accurate analysis results are obtained by optimizing prompt words.

2. The intelligent contract content processing method based on multimodal collaboration according to claim 1, characterized in that, In step S1, the multimodal collaborative parsing includes: Use OCR to efficiently extract regular printed text; Layout analysis is performed using a Transformer model based on the PubLayNet dataset to accurately identify the position of the contract's title hierarchy, paragraph divisions, tables, or seals, and output the spatial coordinates of the identified positions. Use large visual models to directly generate structured descriptions for unconventional printed text such as handwritten signatures, complex tables, and text within seals.

3. The intelligent contract content processing method based on multimodal collaboration according to claim 1, characterized in that, In step S2, the semantic block segmentation includes: The text is segmented according to the contract's clause hierarchy to ensure the semantic integrity of each text block; Each text block is associated with a page number and image coordinates for subsequent precise location of the original image; We construct a dual-vector embedding that includes general semantic vectors and contract-specific vectors, and enhance the semantic matching of professional terms based on contract content corpus.

4. The intelligent contract content processing method based on multimodal collaboration according to claim 1, characterized in that, In step S3, the multi-vector hybrid retrieval strategy includes: Precise keyword matching targets specific terms and identifies highly relevant text blocks. General vector retrieval, using the BGE-M3 model for vector retrieval, covering synonyms or similar expressions; Domain vector retrieval enhances the accuracy of professional semantic matching for industry-specific expressions.

5. The intelligent contract content processing method based on multimodal collaboration according to claim 1, characterized in that, In step S4, the closed-loop verification obtains image coordinates by locating text blocks through hybrid retrieval, crops the original image region, and directly calls the visual big model to parse information from the original image without relying on intermediate OCR results. Based on the retrieved and located coordinates, the visual big model is called to perform parsing to avoid OCR recognition errors.

6. The intelligent contract content processing method based on multimodal collaboration according to claim 1, characterized in that, In step S4, the optimized prompt words are constructed by building a contract domain-specific prompt word template library through few-shot learning. Domain-specific prompt word templates are constructed for different contract types such as leasing, procurement, labor services and loans. Based on domain expert annotation or legal text analysis, key review items and related elements are extracted.

7. The intelligent contract content processing method based on multimodal collaboration according to claim 6, characterized in that, The process involves constructing a dedicated prompt word template library for the contract domain, generating general prompt word templates, designing a general structure based on element parsing requirements, manually modifying the general templates for each type of contract to adapt to domain characteristics, and using LLaMA's meta-generative model combined with a prototype prompting mechanism for training and inference to automatically generate prompt word templates for other review items of similar contracts.

8. The intelligent contract content processing method based on multimodal collaboration according to claim 1, characterized in that, The dynamic optimization of the retrieval strategy includes: The results are reordered by deeply modeling the semantic interaction between candidate text blocks and user queries using a cross-encoder; A hybrid retrieval method using keywords and multi-vectors is employed to quickly recall candidate semantic blocks; Each recalled candidate semantic block is concatenated with the user query to form an input sequence, which is then input into a cross encoder to obtain a relevance score. Reorder candidate blocks from highest to lowest score to filter out low-quality matches.

9. The intelligent contract content processing method based on multimodal collaboration according to claim 8, characterized in that, The cross encoder also includes a dynamic optimization step, which sets a result adjustment function in the contract review interface, allowing users to manually adjust the sorting of search results or mark incorrect matches, automatically records user operation behavior as weak supervision signals, periodically converts weak supervision signals into training samples, updates the cross encoder, and retains historical learning results.

10. The intelligent contract content processing method based on multimodal collaboration according to claim 1, characterized in that, The large visual model is an LLaMA model, which extracts visual features of image regions through a visual encoder, aligns the visual features with text prompts in semantic space through a projection layer, and finally generates structured text parsing results autoregressively by a language decoder.

Images