A method and device for extracting contract information based on a multimodal large language model

Abstract

This invention discloses a method and device for contract information extraction based on a multimodal large language model, belonging to the field of smart contract technology. It addresses the technical problems of existing contract information extraction methods being limited by rule templates, making it difficult to extract graphic information and key named entity information from contracts, resulting in omissions and inflexible extraction processes. The method includes: acquiring contract data in different formats and constructing a model training dataset; constructing a latent space mapping model between contract image data and contract text data; training the latent space mapping model using the model training dataset, and obtaining a general text data encoder and a general image data encoder based on the model validation results; constructing a large language model for contract extraction based on the general text data encoder and the general image data encoder; training the large language model for contract extraction using the model training dataset, and validating the accuracy of information extraction.

Description

Technical Field

[0001] This invention relates to the field of smart contract technology, and in particular to a method and device for extracting contract information based on a multimodal large language model. Background Technology

[0002] Contract information extraction is a crucial step in the wave of enterprise informatization and digitalization. By extracting key information from contracts, the efficiency and accuracy of contract information management can be improved, addressing pain points in traditional contract management, reducing contract risks, strengthening contract control, and enabling contract information sharing, thus providing enterprises with better contract management and operational support. Traditional contract information extraction methods are divided into manual extraction and rule-based extraction. Manual extraction is costly and inefficient, while rule-based extraction is limited by templates.

[0003] With the rapid development of deep learning, the combination of neural networks and rule extraction has become widespread, improving the completeness of information extraction. However, this method is still limited by rule templates, making it difficult to extract chart information from contracts, as well as key named entity information such as company, amount, and time, resulting in omissions and an inflexible extraction process. Summary of the Invention

[0004] This invention provides a method and device for extracting contract information based on a multimodal large language model, which solves the following technical problems: existing contract information extraction methods are limited by rule templates, making it difficult to extract chart information and key named entity information in contracts, resulting in omissions, and the extraction process is inflexible.

[0005] The embodiments of the present invention adopt the following technical solutions:

[0006] On one hand, embodiments of the present invention provide a method for extracting contract information based on a multimodal large language model. The method includes: acquiring contract data in different formats and constructing a model training dataset; wherein the contract data in different formats includes contract image data and contract text data; and the model training dataset includes a first dataset, a second dataset, a third dataset, a fourth dataset, and a fifth dataset.

[0007] Construct a latent space mapping model between contract image data and contract text data;

[0008] The latent space mapping model is trained using the model training dataset, and a general text data encoder and a general image data encoder are obtained based on the model validation results.

[0009] Based on the general text data encoder and the general image data encoder, a large language model for contract extraction is constructed.

[0010] The contract extraction large language model is trained using the model training dataset, and the accuracy of information extraction is verified.

[0011] In one feasible implementation, contract data in different formats is acquired and a model training dataset is constructed, specifically including:

[0012] The contract image data and the contract text data are standardized.

[0013] The standardized contract image data is cropped according to preset cropping rules to obtain contract image cropped data; the standardized contract text data is decomposed according to preset decomposition rules to obtain contract text decomposition data.

[0014] Align the contract image cropping data and the contract text decomposition data to obtain the first dataset;

[0015] Images of a preset area are extracted from the standardized contract image data, and information is extracted to construct a second dataset and a third dataset.

[0016] Prompt words were extracted from standardized contract image data and contract text data to construct the fourth and fifth datasets.

[0017] The first dataset, the second dataset, the third dataset, the fourth dataset, and the fifth dataset are divided into training sets and test sets according to a preset ratio to obtain the model training dataset.

[0018] In one feasible implementation, a preset area image is extracted from the standardized contract image data, and information is extracted to construct a second dataset and a third dataset, specifically including:

[0019] Extract images of the table area and the official seal area from the standardized contract image data;

[0020] Using the PPOCRLabel tool, the image of the table area is labeled as text data in a first preset format with LaTeX as the layout; and the image of the official seal area is labeled as text data in a second preset format containing company information.

[0021] Extract the first Prompt word from the first preset format text data;

[0022] Extract the second Prompt prompt word from the second preset format text data;

[0023] The table area image, the first preset format text data, and the first Prompt prompt are used to construct a second dataset;

[0024] The official seal area image, the second preset format text data, and the second Prompt prompt are used to construct a third dataset.

[0025] In one feasible implementation, prompt words are extracted from the standardized contract image data and contract text data to construct a fourth and fifth dataset, specifically including:

[0026] Extract the third Prompt word from the standardized contract image data and the contract text data;

[0027] The standardized contract image data, the third Prompt message, and the standardized contract text data are used to construct a fourth dataset;

[0028] The standardized contract text data and the third prompt word are used to construct the fifth dataset.

[0029] In one feasible implementation, a latent space mapping model between contract image data and contract text data is constructed, specifically including:

[0030] The contract image data is segmented using Patch, and then Patch encoding and position encoding are performed.

[0031] An image data encoder is constructed using a Transformer Encoder with a self-attention mechanism at its core and a feedforward neural network (FFN) to obtain the latent space feature vector V of the image data. img ;

[0032] The contract text data is tokenized, and then tokenized and positionally encoded.

[0033] A text encoder with pre-trained weights and a feedforward neural network (FFN) are used to construct a text data encoder to obtain the latent space feature vector V of the contract text data. txt ;

[0034] Based on the image data encoder and the text data encoder, a latent space mapping model between contract image data and contract text data is constructed.

[0035] In one feasible implementation, the method further includes:

[0036] L2 regularization is applied to the latent space feature vectors output by the image data encoder and the text data encoder respectively to obtain the cosine similarity formula:

[0037] By applying the 2-norm to the latent space feature vectors output by the image data encoder and the text data encoder respectively, the Tanimoto coefficient formula is obtained:

[0038] Based on the cosine similarity formula and the Tanimoto coefficient formula, the loss function of the latent space mapping model is obtained: Loss = β·Cos(V txt V img )+(1-β)Tan(V txt V img ); where β is the fusion weight of cosine similarity and Tanimoto coefficient, β∈[0,1].

[0039] In one feasible implementation, the latent space mapping model is trained using the model training dataset, and a general text data encoder and a general image data encoder are obtained based on the model validation results, specifically including:

[0040] Input the training set from the first dataset into the latent space mapping model to train the mapping relationship between contract text data and contract image data, and obtain the trained latent space mapping model.

[0041] The test set from the first dataset is input into the latent space mapping model to verify the accuracy of the mapping relationship between contract text data and contract image data. The weights of the text data encoder and image data encoder that meet the accuracy requirements are saved to obtain the general text data encoder and the general image data encoder.

[0042] In one feasible implementation, a large language model for contract extraction is constructed based on the general text data encoder and the general image data encoder, specifically including:

[0043] A data format differentiation module is constructed to divide contract text data and contract image data, and to call the general text data encoder and the general image data encoder.

[0044] A Prompt word encoder is constructed based on the output vector of a general text data encoder or a general image data encoder.

[0045] A base language model is constructed, and the base language model is fine-tuned through the low-rank fine-tuning strategy Lora module, so as to transform the base language model into a professional model suitable for contract information extraction.

[0046] Build a post-processing module to handle chart tasks;

[0047] The data format differentiation module, the general text data encoder, the general image data encoder, the Prompt word encoder, the Lora module, the base large language model, and the post-processing module are combined into the contract information extraction large language model.

[0048] In one feasible implementation, the contract extraction large language model is trained using the model training dataset, and the accuracy of information extraction is verified, specifically including:

[0049] The training sets from the second, third, fourth, and fifth datasets are input into the contract information extraction large language model, and only the prompt word encoder, Lora module, and post-processing module in the contract information extraction large language model are trained.

[0050] Input the test sets from the second, third, fourth, and fifth datasets into the contract information extraction big language model to verify the accuracy of the contract information extraction big language model in extracting the required information from the contract documents.

[0051] On the other hand, embodiments of the present invention also provide a contract information extraction device based on a multimodal large language model, the device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to execute the contract information extraction method based on a multimodal large language model.

[0052] Compared with the prior art, the contract information extraction method and device based on a multimodal large language model provided in this invention have the following beneficial effects:

[0053] This invention addresses the limitations of existing contract information extraction methods, which are restricted by rule templates and struggle to extract graphical information from contracts. Based on a multimodal large language model, this method categorizes and processes textual, image, and tabular information within contracts, and then classifies this information using a multimodal large language model. This significantly improves the efficiency of contract information extraction, enhances the flexibility of information acquisition, and makes contract management more convenient and intelligent. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0055] Figure 1 A flowchart of a contract information extraction method based on a multimodal large language model is provided for an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of a latent space mapping model architecture provided in an embodiment of the present invention;

[0057] Figure 3 A schematic diagram of a Prompt word encoder structure provided in an embodiment of the present invention;

[0058] Figure 4 This is a schematic diagram of a LoRa fine-tuning base large language model structure provided in an embodiment of the present invention;

[0059] Figure 5 This is a schematic diagram of a large language model structure for contract extraction provided in an embodiment of the present invention;

[0060] Figure 6 This is a schematic diagram of the structure of a contract information extraction device based on a multimodal large language model, provided in an embodiment of the present invention. Detailed Implementation

[0061] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0062] This invention provides a method for extracting contract information based on a multimodal large language model, such as... Figure 1 As shown, the contract information extraction method based on a multimodal large language model specifically includes steps S101-S105:

[0063] S101. Obtain contract data in different formats and construct a model training dataset. The model training dataset includes the first dataset, the second dataset, the third dataset, the fourth dataset, and the fifth dataset.

[0064] Specifically, the process begins by acquiring contract data in different formats, including contract image data and contract text data. Then, the contract image data and contract text data undergo standardization processing.

[0065] Furthermore, the standardized contract image data is cropped using preset cropping rules to obtain cropped contract image data; the standardized contract text data is decomposed using preset decomposition rules to obtain decomposed contract text data.

[0066] Furthermore, by aligning the cropped contract image data and the decomposed contract text data, a first dataset is obtained; images of preset regions are extracted from the standardized contract image data, and information is extracted to construct the second and third datasets; prompt words are extracted from the standardized contract image data and contract text data to construct the fourth and fifth datasets.

[0067] Specifically, images of predetermined regions are extracted from the standardized contract image data, and information is extracted to construct a second and a third dataset, which include:

[0068] From the standardized contract image data, extract the table area and the official seal area images; using the PPOCRLabel tool, label the table area images as first preset format text data with LaTeX layout; and label the official seal area images as second preset format text data containing company information; extract the first prompt word from the first preset format text data; extract the second prompt word from the second preset format text data; construct the second dataset from the table area images, the first preset format text data, and the first prompt word. Construct the third dataset from the official seal area images, the second preset format text data, and the second prompt word.

[0069] Clue words were extracted from standardized contract image and text data to construct a fourth and fifth dataset, specifically including:

[0070] The third prompt word is extracted from the standardized contract image data and contract text data; the standardized contract image data, the third prompt word, and the standardized contract text data are used to construct the fourth dataset; the standardized contract text data and the third prompt word are used to construct the fifth dataset.

[0071] Finally, the first, second, third, fourth, and fifth datasets are divided into training and testing sets according to a preset ratio to obtain the model training dataset.

[0072] As a feasible implementation method, the specific steps for constructing the model training dataset are as follows:

[0073] (1-1): Constructing the training dataset for the latent space mapping model TI (Text and Img) of text and image data: First, obtain contract image format data (such as PDF, JPG, PNG, etc.), convert the data formats PDF, PNG, and other data formats to JPG format, and then crop the JPG format image data according to the division of sentences, paragraphs, and chapters; then obtain contract text format data (such as TXT, Word, etc.), convert the data formats Word and other data formats to TXT format, and then decompose the TXT format text data according to the division of sentences, paragraphs, and chapters.

[0074] (1-2): Align the obtained JPG format image data and TXT format text data, use PPOCRLabel as the data annotation tool, and obtain the first dataset D1 = [JPG, TXT].

[0075] (1-3): Construct a large language model training dataset for contract extraction, extract table areas from JPG format image data, label them as TXT format text data with LaTeX format using PPOCRLabel, extract the Prompt prompts around the table information, such as 'parse the information in Table 1 of the contract', and obtain the second dataset D2 = [Table, Prompt, LaTeX].

[0076] (1-4): Extract the official seal area from the JPG format image data, label it with PPOCRLabel as TXT format text data containing the company information, and extract the Prompt message content around the official seal information, such as 'parse the official seal information of Party A in the contract', to obtain the third dataset.

[0077] D3 = [Seal, Prompt, TXT]. If the contract contains images with other information, the processing method is the same as in this step.

[0078] (1-5): For the text portion of the JPG format image data and TXT format text data obtained in the first step, the Prompt prompts are related to tasks such as knowledge extraction and content summarization, such as 'purchase amount', 'terminal summary', 'completion time', etc., resulting in the fourth dataset D4 = [JPG, Prompt, TXT] and the fifth dataset D5 = [TXT, Prompt, TXT].

[0079] (1-6): Divide the datasets D1, D2, D3, D4, and D5 into training and test sets in an 8:2 ratio.

[0080] S102. Construct a latent space mapping model between contract image data and contract text data.

[0081] Specifically, the contract image data is patched and then encoded using patch encoding and positional encoding. Then, a Transformer Encoder with a self-attention mechanism and a feedforward neural network (FFN) is used to construct an image data encoder, obtaining the latent space feature vector V of the contract image data. img .

[0082] Through practice, this invention employs a 4-layer Transformer Encode with a self-attention mechanism at its core and a 1-layer FFN to construct an image data editor, thereby achieving the best image data editing effect.

[0083] Furthermore, the contract text data is tokenized, and then tokenized and positionally encoded. A text encoder with pre-trained weights and a 1-layer FFN is used to construct the text data encoder, obtaining the latent space feature vector V of the contract text data. txt .

[0084] Furthermore, based on the image data encoder and the text data encoder, a latent space mapping model between contract image data and contract text data is constructed.

[0085] Furthermore, L2 regularization is applied to the latent space feature vectors output by the image data encoder and the text data encoder respectively to obtain the cosine similarity formula:

[0086] By applying the 2-norm to the latent space feature vectors output by the image data encoder and the text data encoder respectively, the Tanimoto coefficient formula is obtained:

[0087] Based on the cosine similarity formula and the Tanimoto coefficient formula, the loss function of the latent space mapping model is obtained: Loss = β·Cos(V txt V img )+(1-β)Tan(V txt V img ); where β is the fusion weight of cosine similarity and Tanimoto coefficient, β∈[0,1].

[0088] As a feasible implementation method, Figure 2 This is a schematic diagram of a latent space mapping model architecture provided in an embodiment of the present invention, such as... Figure 2 As shown, the specific steps for constructing the latent space mapping model are as follows:

[0089] (2-1): Drawing inspiration from CLIP contrastive learning, the contract image data is patched, then patch-encoded and position-encoded. A four-layer Transformer Encoder with a self-attention mechanism and a one-layer FFN are then used to construct the image data encoder, yielding the latent space feature vector V of the image data. img .

[0090] (2-2): The text data is tokenized, then tokenized and positionally encoded. A text encoder is constructed using CLIP (with pre-trained weights) and a 1-layer FFN (Freeform Flow Network) to obtain the latent space feature vector V of the text data. txt .

[0091] (2-3): A single-layer FFN in the image data encoder and text data encoder is used to ensure the obtained latent space feature vector V img and V txt They have the same dimensions.

[0092] (2-4): Cosine similarity and Tanimoto coefficient weighted fusion are used as the loss function of the TI model.

[0093] S103. Train the latent space mapping model using the model training dataset, and obtain the general text data encoder and the general image data encoder based on the model validation results.

[0094] Specifically, the training set from the first dataset is input into the latent space mapping model to train the mapping relationship between contract text data and contract image data, thus obtaining the trained latent space mapping model.

[0095] Furthermore, the test set from the first dataset is input into the latent space mapping model to verify the accuracy of the mapping relationship between contract text data and contract image data. The weights of the text data encoder and image data encoder that meet the accuracy requirements are saved to obtain the general text data encoder and the general image data encoder.

[0096] As a feasible implementation method, the training set from the first dataset is fed into the latent space mapping model. A warmup training strategy is used for the learning rate, and the model is trained for 100 epochs to obtain the mapping relationship between the contract text data and the contract image data, resulting in a trained latent space mapping model. Then, the test set from the first dataset is fed into the model to verify the accuracy of the mapping relationship between the contract text data and the contract image data. The weights of the text data encoder and image data encoder that meet the accuracy requirements are saved, resulting in a general text data encoder and image data encoder.

[0097] S104. Based on a general text data encoder and a general image data encoder, a large language model for contract extraction is constructed.

[0098] Specifically, a data format differentiation module is constructed to divide contract text data and contract image data, and to call a general text data encoder and a general image data encoder.

[0099] Furthermore, a Prompt word encoder is constructed based on the output vector of a general text data encoder or a general image data encoder.

[0100] As a feasible implementation method, Figure 3 This is a schematic diagram of a Prompt word encoder structure provided in an embodiment of the present invention, as shown below. Figure 3 As shown, the Prompt encoder is constructed using a two-layer Transformer Encoder with a cross-attention mechanism at its core. In the cross-attention mechanism, Q (query) and K (key) come from the output of a general text data encoder or a general image data encoder, while V (value) comes from the output of the Prompt encoder.

[0101] Furthermore, a base-based large language model is constructed, and the base-based large language model is fine-tuned through the low-rank fine-tuning strategy Lora module, transforming the base-based large language model into a professional model suitable for contract information extraction.

[0102] As a feasible implementation method, Figure 4 This is a schematic diagram of a LoRa fine-tuning base large language model structure provided in an embodiment of the present invention, as shown below. Figure 4 As shown, a low-rank fine-tuning strategy (Lora) is used to fine-tune the base large language model. The dimensionality reduction matrix A is initialized using a random Gaussian distribution, while the dimensionality increase matrix B is initialized with a zero matrix to ensure that the matrix of this additional branch remains a zero matrix in the early stages of training. This lightweight fine-tuning transforms the large model into a specialized model suitable for contract information extraction.

[0103] Furthermore, a post-processing module is built to handle chart tasks.

[0104] The post-processing module is used to handle tasks such as LaTeX and official seal extraction. Plain text is output directly, LaTeX is output as a table or Excel file, and official seals are output as images.

[0105] Finally, the data format differentiation module, general text data encoder, general image data encoder, Prompt word encoder, LoRa module, base large language model, and post-processing module are combined into a large language model for contract information extraction.

[0106] The loss function of the multimodal contract information extraction large language model uses weighted KL divergence and MSE to measure the difference between the generated sample distribution and the real data distribution. The loss function is defined as follows: In the formula, y i For real sample data, To generate sample data, N is the total number of sample data, i is the sample number, i∈[1,N], θ is the dynamic weight, β∈[0,1].

[0107] As a feasible implementation method, Figure 5 This is a schematic diagram of a large language model structure for contract extraction provided in an embodiment of the present invention, such as... Figure 5 As shown, contract text data and contract image data are processed by a data format differentiation module and then input into the contract text data encoder and contract image data encoder, respectively, for encoding. The output vectors are then input into the Prompt encoder for further encoding, and then into the base large language model. The Lora module is used to fine-tune the base large language model, and the processing results are input into the post-processing module to complete information extraction. The module marked with 'snowflake' is the weight freezing module, meaning that this module only performs forward computation and does not perform backpropagation.

[0108] S105. Train a large language model for extracting information using the model training dataset, and verify the accuracy of information extraction.

[0109] Specifically, the training sets from the second, third, fourth, and fifth datasets are input into the contract information extraction large language model, and only the prompt word encoder, Lora module, and post-processing module in the contract information extraction large language model are trained.

[0110] Furthermore, the test sets from the second, third, fourth, and fifth datasets were input into the contract information extraction big language model to verify the accuracy of the contract information extraction big language model in extracting the required information from the contract documents.

[0111] As a feasible implementation method, the training sets from datasets D2, D3, D4, and D5 are fed into the large-scale language model for contract information extraction. Only the prompt encoder, LoRa module, and post-processing module are trained. The learning rate variation strategy is warmup, and the training is performed for 200 epochs to obtain the multimodal large-scale language model for contract information extraction. Then, the test sets from datasets D2, D3, D4, and D5 are fed into the large-scale language model for contract information extraction to verify the accuracy of the multimodal large-scale language model in extracting the required information from contract documents.

[0112] In addition, embodiments of the present invention also provide a contract information extraction device based on a multimodal large language model, such as... Figure 6 As shown, the contract information extraction device based on a multimodal large language model specifically includes:

[0113] At least one processor; and a memory communicatively connected to the at least one processor; wherein,

[0114] The memory stores instructions that can be executed by at least one processor, so that at least one processor can perform the following:

[0115] Acquire contract data in different formats and construct a model training dataset; wherein, the contract data in different formats includes contract image data and contract text data; the model training dataset includes a first dataset, a second dataset, a third dataset, a fourth dataset, and a fifth dataset;

[0116] Construct a latent space mapping model between contract image data and contract text data;

[0117] The latent space mapping model is trained using the model training dataset, and a general text data encoder and a general image data encoder are obtained based on the model validation results.

[0118] Based on the general text data encoder and the general image data encoder, a large language model for contract extraction is constructed.

[0119] The contract extraction large language model is trained using the model training dataset, and the accuracy of information extraction is verified.

[0120] The various embodiments in this invention are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0121] The foregoing has described specific embodiments of the present invention. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0122] The above description is merely an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the embodiments of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for extracting contract information based on a multimodal large language model, characterized in that, The method includes: Acquire contract data in different formats and construct a model training dataset; wherein, the contract data in different formats includes contract image data and contract text data; the model training dataset includes a first dataset, a second dataset, a third dataset, a fourth dataset, and a fifth dataset; Constructing a latent space mapping model between contract image data and contract text data specifically includes: The contract image data is segmented using Patch, and then Patch encoding and position encoding are performed. An image data encoder is constructed using a Transformer Encoder with a self-attention mechanism and a feedforward neural network (FFN) to obtain the latent space feature vectors of the image data. ; The contract text data is tokenized, and then tokenized and positionally encoded. A text encoder with pre-trained weights and a feedforward neural network (FFN) are used to construct a text data encoder to obtain the latent space feature vector of the contract text data. ; Based on the image data encoder and the text data encoder, a latent space mapping model between contract image data and contract text data is constructed. L2 regularization is applied to the latent space feature vectors output by the image data encoder and the text data encoder respectively to obtain the cosine similarity formula: ; By applying the 2-norm to the latent space feature vectors output by the image data encoder and the text data encoder respectively, the Tanimoto coefficient formula is obtained: ; Based on the cosine similarity formula and the Tanimoto coefficient formula, the loss function of the latent space mapping model is obtained: ;in, The fusion weights for cosine similarity and Tanimoto coefficient are... ; The latent space mapping model is trained using the model training dataset, and a general text data encoder and a general image data encoder are obtained based on the model validation results. Based on the aforementioned general text data encoder and general image data encoder, a large language model for contract extraction is constructed, specifically including: A data format differentiation module is constructed to divide contract text data and contract image data, and to call the general text data encoder and the general image data encoder. A Prompt word encoder is constructed based on the output vector of a general text data encoder or a general image data encoder. A base language model is constructed, and the base language model is fine-tuned through the low-rank fine-tuning strategy Lora module, so as to transform the base language model into a professional model suitable for contract information extraction. Build a post-processing module to handle chart tasks; The data format differentiation module, the general text data encoder, the general image data encoder, the Prompt word encoder, the Lora module, the base large language model, and the post-processing module are combined into the contract information extraction large language model. The contract extraction large language model is trained using the model training dataset, and the accuracy of information extraction is verified.

2. The method for extracting contract information based on a multimodal large language model according to claim 1, characterized in that, Acquire contract data in different formats and construct a model training dataset, specifically including: The contract image data and the contract text data are standardized. The standardized contract image data is cropped according to preset cropping rules to obtain contract image cropped data; the standardized contract text data is decomposed according to preset decomposition rules to obtain contract text decomposition data. Align the contract image cropping data and the contract text decomposition data to obtain the first dataset; Images of a preset area are extracted from the standardized contract image data, and information is extracted to construct a second dataset and a third dataset. Prompt words were extracted from standardized contract image data and contract text data to construct the fourth and fifth datasets. The first dataset, the second dataset, the third dataset, the fourth dataset, and the fifth dataset are divided into training sets and test sets according to a preset ratio to obtain the model training dataset.

3. The method for extracting contract information based on a multimodal large language model according to claim 2, characterized in that, Images of predetermined regions are extracted from the standardized contract image data, and information is extracted to construct a second and a third dataset, specifically including: Extract images of the table area and the official seal area from the standardized contract image data; Using the PPOCRLabel tool, the image of the table area is labeled as text data in a first preset format with LaTeX as the layout; and the image of the official seal area is labeled as text data in a second preset format containing company information. Extract the first Prompt word from the first preset format text data; Extract the second Prompt prompt word from the second preset format text data; The table area image, the first preset format text data, and the first Prompt prompt are used to construct a second dataset; The official seal area image, the second preset format text data, and the second Prompt prompt are used to construct a third dataset.

4. The method for extracting contract information based on a multimodal large language model according to claim 2, characterized in that, Clue words were extracted from the standardized contract image data and contract text data to construct the fourth and fifth datasets, specifically including: Extract the third Prompt word from the standardized contract image data and the contract text data; The standardized contract image data, the third Prompt message, and the standardized contract text data are used to construct a fourth dataset. The standardized contract text data and the third prompt word are used to construct the fifth dataset.

5. The method for extracting contract information based on a multimodal large language model according to claim 1, characterized in that, The latent space mapping model is trained using the model training dataset, and a general text data encoder and a general image data encoder are obtained based on the model validation results, specifically including: Input the training set from the first dataset into the latent space mapping model to train the mapping relationship between contract text data and contract image data, and obtain the trained latent space mapping model. The test set from the first dataset is input into the latent space mapping model to verify the accuracy of the mapping relationship between contract text data and contract image data. The weights of the text data encoder and image data encoder that meet the accuracy requirements are saved to obtain the general text data encoder and the general image data encoder.

6. The method for extracting contract information based on a multimodal large language model according to claim 1, characterized in that, The contract extraction large language model is trained using the model training dataset, and the accuracy of information extraction is verified, specifically including: The training sets from the second, third, fourth, and fifth datasets are input into the contract information extraction large language model, and only the prompt word encoder, Lora module, and post-processing module in the contract information extraction large language model are trained. Input the test sets from the second, third, fourth, and fifth datasets into the contract information extraction big language model to verify the accuracy of the contract information extraction big language model in extracting the required information from the contract documents.

7. A contract information extraction device based on a multimodal large language model, characterized in that, The device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, enabling the at least one processor to perform a contract information extraction method based on a multimodal large language model according to any one of claims 1-6.

Images