Intelligent legal document generation method and system based on large model

Abstract

The invention discloses an intelligent legal instrument generation method and system based on a large model, and relates to the technical field of natural language processing. According to the method, law related texts are collected through multiple channels and preprocessed, a law domain knowledge graph is constructed, and after pre-training is conducted through a BERT model, the graph and a document sample are fused for fine tuning training; a user selects a document type and an input parameter by means of the interactive interface and transmits the document type and the input parameter to the model to generate a first draft of the legal document, and then multi-dimensional evaluation and optimization circulation are carried out until the legal document reaching the standard is output; the method covers the whole process of data processing, knowledge graph construction, model training and document generation evaluation, enriches knowledge reserve by using multi-source data, improves model pertinence by combining pre-training and fine tuning, and guarantees document quality through multi-dimensional evaluation. The problems of low legal instrument writing efficiency, unstable quality and the like are effectively solved, the legal instrument generation efficiency and quality are improved, and intelligent development of the legal field is promoted.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and more specifically, to a method and system for intelligently generating legal documents based on a large model. Background Art

[0002] In recent years, artificial intelligence technology has flourished, and deep learning technology has made great progress, greatly promoting the improvement of natural language processing (NLP) capabilities. This has not only improved the results of traditional NLP tasks such as machine translation and text classification, but also created opportunities for emerging application areas such as automated document generation. Especially in the legal industry, due to its strict requirements for precise language expression and standardized text format, there is an urgent need for technology that can automatically generate legal documents that meet specific needs.

[0003] At present, some tools have appeared on the market that attempt to realize the automated production of legal documents through rule engines or simple template filling mechanisms. For example, some primary products allow users to input specific case details based on a pre-set document structure, and then generate preliminary legal documents. However, such methods lack flexibility and adaptability, and are difficult to cope with complex and changing actual business needs. In addition, although some researchers have used shallow neural network models for text generation, which has improved the quality of generated text to a certain extent, the performance in terms of coherence and logical consistency of long texts is unsatisfactory.

[0004] In general, existing legal document automation generation tools either rely too much on fixed rule bases and lack the ability to respond to novel situations; or are based on basic artificial intelligence algorithms and cannot guarantee the professional level and accuracy of the generated documents. These defects limit their widespread application in actual workflows. Summary of the invention

[0005] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a method and system for intelligently generating legal documents based on a large model.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] In a first aspect, a method for intelligently generating legal documents based on a large model comprises the following steps:

[0008] Step 1: Collect legal-related texts from multiple channels and pre-process the collected data;

[0009] Step 2: Perform knowledge extraction, knowledge fusion and disambiguation operations on the pre-processed legal-related texts, and integrate the processed information to obtain a legal field knowledge graph;

[0010] Step 3: Select the BERT model for pre-training, including model initialization, hyperparameter configuration, pre-training task determination, data preparation, and pre-training process;

[0011] Step 4: Merge the legal knowledge graph data with the legal document sample data and adjust the input format, clearly set the fine-tuning training objectives, and input the model in batches for fine-tuning training;

[0012] Step 5: The user selects the document type and inputs relevant parameters through the interactive interface, which are passed to the fine-tuned model. The model determines the draft framework based on the document type and generates the draft of the legal document.

[0013] Step 6: Evaluate the quality of the first draft in multiple dimensions. If optimization is needed, feedback the evaluation results to the model training phase for targeted improvements. Once the quality is achieved, legal documents that meet the requirements will be output to the user.

[0014] Preferably, in step 2, the knowledge extraction includes entity extraction, relationship extraction and attribute extraction; the knowledge fusion and disambiguation includes entity alignment, entity disambiguation and knowledge integration.

[0015] Preferably, in step 3, a BERT-large architecture is selected, including a 24-layer Transformer encoder, a hidden layer size of 1024, and a number of self-attention heads of 16; the learning rate is set between 1e-6 and 3e-6; the training batch size is commonly set to 16 or 32; due to the depth and complexity of the BERT-large model, the number of training rounds is generally set to 10-20 rounds; and the Adam optimizer is selected;

[0016] Design pre-training tasks, namely the masked language model (MLM) task and the next sentence prediction (NSP) task;

[0017] Collect large-scale general text data as pre-training materials and pre-process these data;

[0018] The preprocessed general text data is input into the pre-training model in sequence according to the set batch size. For each batch of data, the model performs forward propagation calculation according to the set pre-training task, specifically:

[0019] For the MLM task, the cross entropy loss function is used to calculate the cross entropy loss between the probability distribution of the predicted word and the one-hot encoding of the real word; assuming that the vocabulary size is V, the number of masked words is N, and the predicted probability of the jth word in the vocabulary for the i-th masked word is p ij , the true label is y ij , then the loss function formula of the MLM task is:

[0020]

[0021] Among them, the true label y ij Only the position corresponding to the real word is 1, and the rest are 0;

[0022] For the NSP task, the cross entropy loss function is also used to calculate the cross entropy loss between the predicted sentence relationship probability and the true label; assuming that the probability of predicting two sentences to be adjacent is p and the true label is y, the loss function of the NSP task is:

[0023] Loss NSP =-[ylog(p)+(1-y)log(1-p)]

[0024] Among them, the true label y is 1 when it means adjacent, and 0 when it means non-adjacent;

[0025] Calculate the total loss Loss1 = Loss MLM +Loss NSP ;

[0026] Use the optimization algorithm Adam to perform back propagation and adjust the model parameters so that the model is gradually optimized in the direction of reducing the loss value and better completing the pre-training task;

[0027] For the selected Adam optimizer, it updates the parameters based on the first-order moment estimate and second-order moment estimate of the gradient and the set learning rate; its update formula is:

[0028] m t =β1m t-1 +(1-β1)g t

[0029]

[0030]

[0031]

[0032] Among them, m t and v t are the values ​​of the first-order moment estimate and the second-order moment estimate at step t, respectively, t is the gradient of the tth step, α is the learning rate, β1, β2, ε are the hyperparameters of the optimizer, and is the modified first-order moment estimate and second-order moment estimate, θ t is the value of the model parameter at step t, θ t+1 is the updated parameter value;

[0033] Multiple batches of data are repeatedly trained according to the set number of training rounds.

[0034] Preferably, in step 4, key information including entities, relationships and attributes is extracted from the constructed legal field knowledge graph, and the extracted entities and relationships are vectorized;

[0035] For each legal document sample, traverse the text content therein and identify the words or phrases corresponding to the entities in the knowledge graph; merge the identified entity vectors into the text vector sequence of the legal document sample;

[0036] Determine the maximum input length based on input requirements; add special tags to the fused input vector sequence to assist the model in understanding the text structure and task requirements;

[0037] The goal setting of fine-tuning training includes the goal of generating legal document format, the goal of accuracy in the use of legal terminology, and the goal of constructing legal logical relationships;

[0038] The legal document samples that integrate legal knowledge graph data and adjust the format are input into the model in batches according to the set size;

[0039] After the data is input into the model, the pre-trained part is first encoded. Through the self-attention mechanism of the multi-layer Transformer encoder, the model can fully capture the semantic features of the text;

[0040] The encoded semantic features will be passed to the generation layer for generating specific document types and the format judgment layer for judging the correctness of the document format for further processing and analysis;

[0041] According to the model output and the real legal document samples, the loss value is calculated according to the set loss function. The loss function includes the basic language generation loss, the legal term loss penalty item, and the legal logic relationship loss penalty item. Specifically, it is:

[0042] Based on language generation loss: For the language part generated by legal documents, the cross entropy loss function is used; let the word probability distribution predicted by the model be p(yx), p mnk represents the probability of predicting the kth word in the vocabulary for the mth sample and the nth word position, y mnk The mth sample and the nth word position predict the true label of the kth word in the vocabulary. The batch size is B, the sample length is S, and the vocabulary size is V. The cross entropy loss formula for the language generation part is:

[0043]

[0044] Where k is the index of the vocabulary dimension;

[0045] Legal term loss penalty: set as the weight coefficient ω of legal term loss termTo emphasize the correct use of legal terms, a penalty term is added; define the function f tern (x) Calculate the number of legal term errors in the mth sample; assuming that there are a total of T legal terms to be checked, the legal term loss penalty formula is:

[0046]

[0047] Legal logic relationship loss penalty item: For the legal logic relationship part, a penalty item is also added; set as the weight coefficient of the legal logic relationship loss ω logic , define the function f logic (x) Calculate the degree of legal logic relationship error in the mth sample; assuming that there are a total of R legal logic relationships to be checked, the legal logic relationship loss penalty formula is:

[0048]

[0049] Total loss function: L2 = L lang +L term +L logic ;

[0050] The back-propagation algorithm is used to calculate the gradient of the loss to the model parameters. Starting from the last layer, the gradient value of each layer parameter is calculated forward in sequence through the chain rule to obtain the gradient vector corresponding to each parameter; the Adam optimizer is used to update the model parameters.

[0051] Preferably, in step 5, the user selects the document type and inputs relevant parameters through the friendly interactive interface provided by the system, and the interface transmits the information to the fine-tuned large model;

[0052] The model determines the draft framework based on the pre-learned structural template and legal logic of this type of document, and generates a draft of the legal document based on legal knowledge and language skills.

[0053] Preferably, in step 6, the quality of the first draft of the legal document is evaluated from the dimensions of language fluency, legal terminology accuracy, logical rationality, and format standardization;

[0054] According to the evaluation results, the draft of the legal document that needs to be optimized and its corresponding evaluation data are collected and marked as training data, and input into the model for fine-tuning training;

[0055] The legal documents generated after optimization and model retraining are evaluated again. When the quality meets the standards, they are output in the format required by the user, and the generation and optimization process data are recorded.

[0056] The second aspect is a legal document intelligent generation system based on a large model, including a data collection and preprocessing module, a legal knowledge graph construction module, a model pre-training module, a model fine-tuning training module, a legal document generation module, and an evaluation and optimization module;

[0057] The data collection and preprocessing module collects legal-related texts through multiple channels, and cleans, normalizes, and classifies and annotates the collected data;

[0058] The legal knowledge graph construction module performs knowledge extraction, knowledge fusion and disambiguation operations on legal texts, and integrates the processed information to obtain a legal field knowledge graph;

[0059] The model pre-training module performs a pre-training operation on the selected BERT-large architecture model;

[0060] The model fine-tuning training module fine-tunes the model for the legal document generation task;

[0061] The legal document generation module receives the document type and related parameter information input by the user and passes them to the model, and generates a first draft of the legal document based on the document structure template and legal logic learned by the model;

[0062] The evaluation and optimization module evaluates the quality of the first draft of the legal document in multiple dimensions, optimizes the model based on the evaluation results and outputs qualified documents, while recording the generation and optimization process data.

[0063] Compared with the prior art, the present invention has the following beneficial effects:

[0064] 1. In the present invention, by collecting legal-related texts through multiple channels, it can widely cover legal materials from different sources and different fields, ensuring the richness and comprehensiveness of the data. From authoritative official legal databases to professional legal document libraries, well-known legal academic resource platforms, and documents issued by government departments, these multi-source data become standardized and orderly after pre-processing, providing high-quality materials for subsequent links, avoiding the tediousness of manual collection and sorting, saving a lot of time and energy, and improving overall work efficiency.

[0065] 2. In the present invention, the legal knowledge graph constructed by knowledge extraction, fusion and disambiguation operations integrates the scattered legal knowledge and presents the complex legal relations, entities and other information in a structured manner. When generating documents, the model can quickly and accurately obtain the required knowledge from them, making the content of the documents more accurate and professional, improving the quality of the documents and reducing the problems caused by lack of knowledge or incorrect citations.

[0066] 3. In the present invention, by selecting the BERT-large architecture for pre-training, its powerful language processing capabilities combined with carefully designed pre-training tasks (such as MLM and NSP tasks) enable the model to master the semantics, grammar and logical relationships of the language well, laying the foundation for the subsequent integration of legal knowledge. In the fine-tuning training stage, goals such as legal document format generation, accuracy in the use of legal terminology and construction of legal logical relationships are set, and a loss function containing corresponding penalty terms is used to encourage the model to focus on learning and accurately apply legal expertise, ensuring that the generated documents are standardized in format, accurate in the use of terminology, and rigorous in logic, showing a high degree of professionalism.

[0067] 4. In the present invention, users only need to operate in a friendly interactive interface, select the document type and input key parameters, and the system can quickly determine the draft framework and generate preliminary legal documents based on the large model that has been pre-trained and fine-tuned. Compared with traditional manual writing, the document generation time is greatly shortened, especially for some common legal documents with relatively fixed formats and regular contents, the efficiency improvement is more significant.

[0068] 5. In the present invention, a multi-dimensional quality assessment mechanism can comprehensively and meticulously check the language fluency, legal terminology accuracy, logical rationality, and format standardization of the first draft, promptly identify problems and feed back relevant data to the model for targeted fine-tuning training. After multiple iterations, the quality of the document is continuously optimized to ensure that the final output document meets high-standard legal requirements, effectively reducing the possibility of subsequent legal risks caused by poor quality of the document. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A flowchart of a method for intelligently generating legal documents based on a large model is proposed for the present invention;

[0070] Figure 2 A flow chart of a legal document intelligent generation system based on a large model is proposed for the present invention. DETAILED DESCRIPTION

[0071] The embodiment further illustrates the method and system for intelligently generating legal documents based on a large model proposed by the present invention.

[0072] Reference Figure 1 , a method for intelligently generating legal documents based on a large model, comprising the following steps:

[0073] Step 1: Collect legal-related texts from multiple channels and pre-process the collected data;

[0074] Step 2: Perform knowledge extraction, knowledge fusion and disambiguation operations on the pre-processed legal-related texts, and integrate the processed information to obtain a legal field knowledge graph;

[0075] Step 3: Select the BERT model for pre-training, including model initialization, hyperparameter configuration, pre-training task determination, data preparation, and pre-training process;

[0076] Step 4: Merge the legal knowledge graph data with the legal document sample data and adjust the input format, clearly set the fine-tuning training objectives, and input the model in batches for fine-tuning training;

[0077] Step 5: The user selects the document type and inputs relevant parameters through the interactive interface, which are passed to the fine-tuned model. The model determines the draft framework based on the document type and generates the draft of the legal document.

[0078] Step 6: Evaluate the quality of the first draft in multiple dimensions. If optimization is needed, feedback the evaluation results to the model training phase for targeted improvements. Once the quality is achieved, legal documents that meet the requirements will be output to the user.

[0079] In the step 1, legal-related texts are collected from official legal databases, professional legal document libraries, well-known legal academic resource platforms, and original texts of laws, regulations, and policy documents issued by government departments; the data preprocessing includes data cleaning, text normalization, and classification and annotation.

[0080] The data cleaning described herein: removes duplicate content in the text to avoid interference of redundant data on model training; identifies and deletes format errors, garbled characters, and obviously incomplete text fragments to improve the quality and availability of the data.

[0081] The text normalization: unifies the text encoding format into the universal UTF-8 for easy processing and storage; standardizes the use of punctuation marks, such as converting full-width punctuation marks into half-width punctuation marks, and corrects punctuation anomalies caused by input errors or non-standardization; corrects common typos and grammatical errors to make the text more in line with language standards, which is conducive to subsequent natural language processing operations.

[0082] The classification annotation described above: classify and organize the text according to the legal field classification (such as civil law, criminal law, administrative law, etc.) so as to carry out subsequent analysis and processing according to the characteristics of different fields; annotate the key elements in the legal text, for example, mark the legal subjects (names of the parties, names of legal persons, etc.), legal acts (signing contracts, filing lawsuits, torts, etc.), legal relationships (rights and obligations in contracts, causal relationships in tort cases, etc.) and the cited legal clause numbers or names, etc., to provide more accurate information for knowledge extraction and model training.

[0083] In the step 2, the knowledge extraction includes entity extraction, relationship extraction and attribute extraction; the knowledge fusion and disambiguation includes entity alignment, entity disambiguation and knowledge integration;

[0084] Entity extraction: Use named entity recognition technology, combined with legal professional dictionaries and grammatical rules, to accurately identify various entities from legal texts; in addition to common names of people, places, and organizations, focus on identifying specific legal entities, such as legal article numbers, legal procedure names (filing, appeal, retrial, etc.), legal liability types (breach of contract, tort liability, etc.), legal document types (contracts, wills, judgments, etc.), etc.; for example, in a contract text, identify "Party A" and "Party B" as contract party entities, "contract signing date" as a time entity, and "subject matter" as a specific contract-related entity.

[0085] Relationship extraction: Based on dependency syntactic analysis and semantic role labeling technology, the relationship between entities is determined by analyzing sentence structure and semantics; for example, in a litigation case text, the "litigation confrontation" relationship between "plaintiff" and "defendant", the "trial" relationship between "court" and "case", and the "inclusion" relationship between "contract" and "clause" are determined. For some complex legal relationships, such as the reference relationship between legal provisions and the hierarchical relationship between legal concepts, they can be determined with the help of special legal knowledge parsing tools or manual auxiliary labeling.

[0086] Attribute extraction: For identified entities, extract relevant attribute information; for example, for a "natural person" entity, extract attributes such as age, gender, occupation, and ID number; for a "contract" entity, extract attributes such as contract amount, contract term, and place of performance; for a "legal provision" entity, extract attributes such as the promulgation time, number of revisions, and scope of application of the provision to further enrich the entity's description information.

[0087] Entity alignment: Since different data sources may have multiple ways of expressing the same entity, for example, "People's Republic of China" may be abbreviated as "China", "Limited Liability Company" may be abbreviated as "Co., Ltd.", these entities with the same name are merged and aligned by calculating the entity name similarity, context-based semantic similarity, and using external authoritative knowledge bases (such as business registration information, government agency directories, etc.) for verification to ensure that each entity in the knowledge graph has a unique identifier.

[0088] Entity disambiguation: For entities with the same name but different actual entities, such as companies with the same name in different regions or legal concepts with the same name in different legal contexts, the entity's attribute information, contextual context, and related domain knowledge are used to distinguish and disambiguate. For example, in a commercial dispute case involving companies in multiple regions, the specific "XX Company" entity is determined by the company's registered address, business scope, and other attributes to avoid entity confusion in the knowledge graph.

[0089] Knowledge integration: Integrate the extracted, fused and disambiguated entity, relationship and attribute information to construct a structured legal knowledge graph; use a graph database (such as Neo4j) for storage, with entities as nodes in the graph, relationships as edges connecting nodes, and attributes as additional information of nodes, so as to intuitively display the network structure and logical relationship between legal knowledge; for example, in the knowledge graph, the "sales contract" node is connected to the "buyer" and "seller" nodes through the "contract party" edge, and the "buyer" node displays its specific information through attribute edges such as "name" and "address". In this way, the internal connection of legal knowledge is clearly presented, providing strong support for subsequent models to use knowledge graphs for reasoning and generate legal documents.

[0090] In step 3, the BERT-large architecture is selected to include a 24-layer Transformer encoder, a hidden layer size of 1024, a number of self-attention heads of 16, a learning rate set between 1e-6 and 3e-6, and a linear decay method is used to adjust the learning rate; due to the depth and complexity of the BERT-large model, it can generally be set to about 10-20 rounds to start the trial; and the Adam optimizer is selected.

[0091] Design pre-training tasks, namely the masked language model MLM task and the next sentence prediction NSP task; when performing the masked language model MLM task, accurately extract the output vector corresponding to the masked position, and convert it into a probability distribution on the vocabulary through a classification layer for predicting the masked word; for the next sentence prediction NSP task, accurately extract the output vector corresponding to the [CLS] tag, and obtain the binary classification probability distribution of the sentence relationship through a linear layer and activation function to determine whether two sentences are adjacent.

[0092] Collect large-scale general text data as pre-training materials and pre-process these data;

[0093] The general text data can come from a wide range of sources, including news articles, blogs, novels, academic papers, and other types of text, to ensure that the model can learn rich and diverse language expressions and semantic information. For example, text data can be collected from news websites, literary websites, academic databases, and other platforms on the Internet, and the amount of data usually reaches billions or even tens of billions of words.

[0094] The preprocessing includes data cleaning, word segmentation, and conversion into the input format required by the model;

[0095] Data cleaning: remove noise data in the text, such as HTML tags, special characters (such as non-printing characters, garbled characters, etc.), duplicate data, etc.; for example, use regular expressions to match and delete HTML tags, filter out control characters and garbled characters in the text, and use data duplication detection algorithms to remove duplicate text paragraphs or sentences;

[0096] Word segmentation: Segment the text into words or word blocks. For English text, you can use common English word segmentation tools, such as the word segmentation function in the NLTK library, to segment the text into word sequences. For Chinese text, you can use Chinese word segmentation tools, such as Jieba word segmentation, to segment the text into word sequences.

[0097] Convert to the input format required by the model: According to the requirements of the BERT model, tokenize the text after word segmentation and add special tags such as [CLS] (indicating the beginning of the text) and [SEP] (indicating the end of the sentence); at the same time, convert the text into a tensor format that the model can process, such as mapping words or word blocks to corresponding index sequences, and truncate or pad the text according to the set maximum input length to ensure that the length of all input texts is consistent.

[0098] The preprocessed general text data is input into the pre-training model in sequence according to the set batch size. For each batch of data, the model performs forward propagation calculation according to the set pre-training task, specifically:

[0099] For the MLM task, the cross entropy loss function is used to calculate the cross entropy loss between the probability distribution of the predicted word and the one-hot encoding of the real word; assuming that the vocabulary size is V, the number of masked words is N, and the predicted probability of the jth word in the vocabulary for the i-th masked word is p ij , the true label is y ij , then the loss function formula of the MLM task is:

[0100]

[0101] Among them, the true label y ij Only the position corresponding to the real word is 1, and the rest are 0;

[0102] For the NSP task, the cross entropy loss function is also used to calculate the cross entropy loss between the predicted sentence relationship probability and the true label; assuming that the probability of predicting two sentences to be adjacent is p and the true label is y, the loss function of the NSP task is:

[0103] Loss NSP =-[ylog(p)+(1-y)log(1-p)]

[0104] Among them, the true label y is 1 when it means adjacent, and 0 when it means non-adjacent;

[0105] Calculate the total loss Loss1 = Loss MLM +Loss NSP .

[0106] Use the optimization algorithm Adam to perform back propagation and adjust the model parameters so that the model is gradually optimized in the direction of reducing the loss value and better completing the pre-training task;

[0107] For the selected Adam optimizer, it updates the parameters based on the first-order moment estimate and second-order moment estimate of the gradient and the set learning rate; its update formula is:

[0108] m t =β1m t-1 +(1-β1)g t

[0109]

[0110]

[0111]

[0112] Among them, m t and v t are the values ​​of the first-order moment estimate and the second-order moment estimate at step t, respectively, t is the gradient of the tth step, α is the learning rate, β1, β2, ε are the hyperparameters of the optimizer, and is the modified first-order moment estimate and second-order moment estimate, θ t is the value of the model parameter at step t, θ t+1 is the updated parameter value.

[0113] Multiple batches of data are repeatedly trained according to the set number of training rounds.

[0114] In step 4, key information including entities, relationships and attributes is extracted from the constructed legal domain knowledge graph, and the extracted entities and relationships are vectorized;

[0115] For example, for a knowledge graph related to a contract, extract the information such as the two parties to the contract (entities), the contract signing relationship (relationship), the registered address and contact information (attributes) of the parties; use the word vector representation method (such as Word2Vec, GloVe or word vectors generated by pre-trained models such as BERT) to convert the extracted entities and relations into vector form; suppose the entity "Party A" is vectorized to obtain the vector e 甲方, the relationship "contract signing" is transformed into vector r 合同签订 .

[0116] For each legal document sample, traverse the text content therein and identify the words or phrases corresponding to the entities in the knowledge graph; merge the identified entity vectors into the text vector sequence of the legal document sample;

[0117] For example, in a contract document, when the entity "Party A" is identified and matched with the corresponding entity in the knowledge graph, the vector e of the "Party A" entity is 甲方 Fusion into the vector sequence near the position where "Party A" appears in the text; direct concatenation, weighted summation and other fusion strategies can be used; if direct concatenation is used, the text vector sequence of the legal document sample at a certain position is s = [v1, v2, ..., v n ], when the "Party A" entity is identified in v i When the positions correspond, the fused vector sequence becomes s'=[v1,v2,…,v i-1 ,e 甲方 ,v i ,…,v n ].

[0118] Determine the maximum input length based on input requirements; add special tags to the fused input vector sequence to assist the model in understanding the text structure and task requirements;

[0119] The maximum input length L is determined according to the model input requirements and the characteristics of hardware resources and training data. For example, it is set to 512 vector unit lengths (the specific length can be determined according to actual experiments and adjustments); special tags are added to the fused input vector sequence. The [CLS] tag is added at the beginning to represent the global features of the entire text, and the [SEP] tag is added between different sentences or paragraphs to distinguish semantic units, helping the model to better understand the text structure and task requirements; for example, the fused vector sequence eventually becomes s final =[[CLS],e 甲方 ,v1,v2,…,[SEP],v i+1 ,…,v n ] If the length of the vector sequence is less than L, a padding operation (such as padding with a zero vector) is performed to make its length reach L.

[0120] The goal setting of fine-tuning training includes the goal of generating legal document format, the goal of accuracy in the use of legal terminology, and the goal of constructing legal logical relationships;

[0121] Legal document format generation goal: Develop detailed format templates for different types of legal documents (such as contracts, complaints, judgments, etc.), requiring the model to accurately generate fixed format content at the beginning of the corresponding document (such as the title, number, signing date, party information, etc. of the contract, the document name, basic information of the plaintiff and defendant, etc. of the complaint), reasonably arrange various clauses or statements in the main body (such as the specific clauses of the contract, the facts and reasons of the complaint, etc.), and meet the standard requirements at the end (such as the signature and seal column of the contract, the signature of the complaint, etc.);

[0122] Legal terminology accuracy goal: to build a comprehensive legal terminology database, covering various commonly used professional terms in the legal field and their accurate interpretations, usage examples, etc.; by setting appropriate weight coefficients ω term (such as ω term =0.5), add a legal term loss penalty term to the loss function. When legal term errors appear in the text generated by the model (such as misusing "deposit" as "deposit", etc.), this term will increase the loss value, prompting the model to pay attention to and use legal terminology accurately;

[0123] Legal logic relationship construction goal: formulate a series of legal logic rules (such as the rules of equivalence of rights and obligations in contracts, the rules of reasonable deduction between litigation requests and factual basis and legal basis in litigation documents, etc.), and set the corresponding weight coefficient ω logic (such as ω logic =0.3) adds a legal logic relationship loss penalty term to the loss function. When there are logical relationship errors in the text generated by the model (such as mismatch between rights and obligations, lack of reasonable basis for litigation requests, etc.), this penalty will guide the model to construct content that conforms to legal logic.

[0124] The legal document samples that have been integrated with the legal field knowledge graph data and adjusted in format are input into the model in sequence according to the set batch size (such as 16 or 32, etc., which should be reasonably selected based on hardware resources and data volume).

[0125] After the data is input into the model, the pre-trained part is first encoded. Through the self-attention mechanism of the multi-layer Transformer encoder, the model can fully capture the semantic features of the text.

[0126] The encoded semantic features will be passed to the generation layer for generating specific document types and the format judgment layer for judging the correctness of the document format for further processing and analysis.

[0127] According to the model output and the real legal document samples, the loss value is calculated according to the set loss function. The loss function includes the basic language generation loss, the legal term loss penalty item, and the legal logic relationship loss penalty item. Specifically, it is:

[0128] Based on language generation loss: For the language part generated by legal documents, the cross entropy loss function is used; let the word probability distribution predicted by the model be p(yx), p mnk represents the probability of predicting the kth word in the vocabulary for the mth sample and the nth word position, y mnk The mth sample and the nth word position predict the true label of the kth word in the vocabulary (usually in the form of one-hot encoding), the batch size is B, the sample length is S, and the vocabulary size is V. The cross entropy loss formula for the language generation part is:

[0129]

[0130] Where k is the index of the vocabulary dimension;

[0131] Legal term loss penalty: set as the weight coefficient ω of legal term loss term To emphasize the correct use of legal terms, a penalty term is added; define the function f tern (x) Calculate the number of legal term errors in the mth sample (determined by comparing with the correct legal term annotations); assuming that there are a total of T legal terms to be checked, the legal term loss penalty formula is:

[0132]

[0133] Legal logic relationship loss penalty item: For the legal logic relationship part, a penalty item is also added; set as the weight coefficient of the legal logic relationship loss ω logic , define the function f logic (x) Calculate the degree of legal logic relationship error in the mth sample (determined by rule checking, logical template matching, etc.); assuming that there are a total of R legal logic relationships to be checked, the legal logic relationship loss penalty term formula is:

[0134]

[0135] Total loss function: L2 = L lang +L term +L logic .

[0136] Use the back-propagation algorithm to calculate the gradient of the loss to the model parameters. Starting from the last layer, the gradient value of each layer parameter is calculated forward in sequence through the chain rule to obtain the complete gradient vector corresponding to each parameter. For example, for a layer parameter θ in the model, the gradient calculation process is as follows: First, calculate the gradient of the total loss L2 to the output of the last layer Where o is the output of the last layer), and then calculate the gradient of the weight matrix according to the activation function and weight matrix W of this layer And the gradient of the bias vector b And so on, the gradient of each layer's parameters is calculated from the last layer forward.

[0137] The Adam optimizer is used to update the model parameters according to the gradient vector and the set learning rate and other parameters. According to the update formula of the Adam optimizer (as described in the pre-training section), m is estimated based on the first-order moment of the gradient. t and the second moment estimate v t And the learning rate α updates the model parameters Through multiple rounds of fine-tuning training, the performance of the model in the task of legal document generation is gradually improved, enabling it to generate high-quality legal documents that comply with legal regulations and user needs.

[0138] In step 5, the user selects the document type and inputs relevant parameters through the friendly interactive interface provided by the system, and the interface transmits the information to the fine-tuned large model;

[0139] The model determines the draft framework based on the pre-learned structural template and legal logic of this type of document, and generates a draft of the legal document based on legal knowledge and language skills.

[0140] In step 6, the quality of the first draft of the legal document is evaluated from the dimensions of language fluency, legal terminology accuracy, logical rationality, and format standardization;

[0141] Language fluency: The language model in natural language processing technology is used to calculate the perplexity index of the text to evaluate language fluency; let the text be T = [w1,w2,…,w n ], where w i represents the i-th word, and the language model calculates the word probability distribution as P(w i w 1；i-1 ), then the calculation formula of perplexity is: The lower the perplexity, the more natural and fluent the text is in terms of language expression, and it conforms to normal language habits and language rules;

[0142] Legal terminology accuracy: Compare all the words in the first draft with the pre-built legal terminology database; let E be the set of legal terms in the first draft, C be the set of correct legal terms, and f term (e) is a function to determine whether the term e is correct (if it is correct, then f term (e)=0, if it is wrong, then f term (e) = 1), then the number of legal term errors N term The calculation formula is:

[0143] Logical rationality: Logical rationality is evaluated by combining a logical rule checker and a legal reasoning model; let R be the set of logical relations in the text, f logic (r) is a function to determine whether the logical relationship r is reasonable (if it is reasonable, then f logic (r)=0, if it is unreasonable, then f logic (r)=1), then the number of legal logical relationship errors N logic The calculation formula is:

[0144] Format standardization: Matching check is performed based on the standard format templates of different types of legal documents; let F be the set of format elements in the first draft, C f is the set of format elements in the correct format template, f format (f) is a function to determine whether the format element f is correct (if it is correct, f format (f)=0, if it is wrong, f format (f) = 1), then the number of format errors N format The calculation formula is:

[0145] According to the evaluation results of the above dimensions, a comprehensive quality score is generated for the first draft by using weighted average and other methods; the weights of the four dimensions of language fluency, legal terminology accuracy, logical rationality, and format standardization are set as w and w respectively. lang =0.2, w term =0.3, w logic =0.3, w format =0.2 (the weight can be adjusted according to actual needs and priorities). After calculating the corresponding scores based on the evaluation indicators of each dimension, the comprehensive quality score S is obtained by weighted summation according to the weights. Assuming that the language fluency score S lang , Legal Terminology Accuracy Score lang , Logical rationality score S lang , Format Standardization Score lang The comprehensive quality score is calculated by the following formula: S = w lang S lang +w term S term +w logic S logic +w format S format .

[0146] Based on the evaluation results, the draft legal documents that need to be optimized and their corresponding evaluation data are collected and marked as training data, and then input into the model for fine-tuning training.

[0147] The legal documents generated after optimization and model retraining are evaluated again. When the quality meets the standards, they are output in the format required by the user, and the generation and optimization process data are recorded.

[0148] Reference Figure 2 , a legal document intelligent generation system based on a large model, including a data collection and preprocessing module, a legal knowledge graph construction module, a model pre-training module, a model fine-tuning training module, a legal document generation module, and an evaluation and optimization module;

[0149] The data collection and preprocessing module collects legal-related texts through multiple channels, and cleans, normalizes, and classifies and annotates the collected data;

[0150] The legal knowledge graph construction module performs knowledge extraction, knowledge fusion and disambiguation operations on legal texts, and integrates the processed information to obtain a legal field knowledge graph;

[0151] The model pre-training module performs a pre-training operation on the selected BERT-large architecture model;

[0152] The model fine-tuning training module fine-tunes the model for the legal document generation task;

[0153] The legal document generation module receives the document type and related parameter information input by the user and passes them to the model, and generates a first draft of the legal document based on the document structure template and legal logic learned by the model;

[0154] The evaluation and optimization module evaluates the quality of the first draft of the legal document in multiple dimensions, optimizes the model based on the evaluation results and outputs qualified documents, while recording the generation and optimization process data.

[0155] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A method for intelligently generating legal documents based on a large model, characterized in that: The following steps are involved: Step 1: Collect legal-related texts through multiple channels and pre-process the collected data; Step 2: Perform knowledge extraction, knowledge fusion and disambiguation operations on the pre-processed legal-related texts, and integrate the processed information to obtain a legal field knowledge graph; Step 3: Select the BERT model for pre-training, including model initialization, hyperparameter configuration, pre-training task determination, data preparation, and pre-training process; Step 4: Merge the legal knowledge graph data with the legal document sample data and adjust the input format, clearly set the fine-tuning training objectives, and input the model in batches for fine-tuning training; Step 5: The user selects the document type and inputs relevant parameters through the interactive interface, which are passed to the fine-tuned model. The model determines the draft framework based on the document type and generates the draft of the legal document. Step 6: Evaluate the quality of the first draft in multiple dimensions. If optimization is needed, feedback the evaluation results to the model training phase for targeted improvements. Once the quality is achieved, legal documents that meet the requirements will be output to the user.

2. According to the method of intelligently generating legal documents based on a large model according to claim 1, it is characterized in that: In the step 1, legal-related texts are collected from official legal databases, professional legal document libraries, well-known legal academic resource platforms, and original texts of laws, regulations, and policy documents issued by government departments; the data preprocessing includes data cleaning, text normalization, and classification and annotation.

3. According to claim 2, a method for intelligently generating legal documents based on a large model is characterized in that: In step 2, the knowledge extraction includes entity extraction, relationship extraction and attribute extraction; the knowledge fusion and disambiguation includes entity alignment, entity disambiguation and knowledge integration.

4. According to claim 3, a method for intelligently generating legal documents based on a large model is characterized in that: In step 3, the BERT-large architecture is selected to include a 24-layer Transformer encoder, a hidden layer size of 1024, and a number of self-attention heads of 16; the learning rate is set between 1e-6 and 3e-6; the training batch size is commonly set to 16 or 32; due to the depth and complexity of the BERT-large model, the number of training rounds is generally set to 10-20 rounds; Select Adam optimizer; Design pre-training tasks, namely the masked language model (MLM) task and the next sentence prediction (NSP) task; Collect large-scale general text data as pre-training materials and pre-process these data; The preprocessed general text data is input into the pre-training model in sequence according to the set batch size. For each batch of data, the model performs forward propagation calculation according to the set pre-training task, specifically: For the MLM task, the cross entropy loss function is used to calculate the cross entropy loss between the probability distribution of the predicted word and the one-hot encoding of the real word; assuming that the vocabulary size is V, the number of masked words is N, and the predicted probability of the jth word in the vocabulary for the i-th masked word is p ij , the true label is y ij , then the loss function formula of the MLM task is: Among them, the true label y ij Only the position corresponding to the real word is 1, and the rest are 0; For the NSP task, the cross entropy loss function is also used to calculate the cross entropy loss between the predicted sentence relationship probability and the true label; assuming that the probability of predicting two sentences to be adjacent is p and the true label is y, the loss function of the NSP task is: Loss NSP =-[ylog(p)+(1-y)log(1-p)] Among them, the true label y is 1 when it means adjacent, and 0 when it means non-adjacent; Calculate the total loss Loss1 = Loss MLM +Loss NSP ; The optimization algorithm Adam is used for back propagation to adjust the parameters of the model so that the model is gradually optimized in the direction of reducing the loss value and better completing the pre-training task. For the selected Adam optimizer, it will update the parameters according to the first-order moment estimate and second-order moment estimate of the gradient and the set learning rate. The update formula is: m t =β1m t-1 +(1-β1)g t Among them, m t and v t are the values ​​of the first-order moment estimate and the second-order moment estimate at step t, respectively, t is the gradient of the tth step, α is the learning rate, β1, β2, ε are the hyperparameters of the optimizer, and is the modified first-order moment estimate and second-order moment estimate, θ t is the value of the model parameter at step t, θ t+1 is the updated parameter value; Multiple batches of data are repeatedly trained according to the set number of training rounds.

5. According to the method for intelligently generating legal documents based on a large model as described in claim 4, it is characterized in that: In step 4, key information including entities, relationships and attributes is extracted from the constructed legal domain knowledge graph, and the extracted entities and relationships are vectorized; For each legal document sample, traverse the text content therein and identify the words or phrases corresponding to the entities in the knowledge graph; merge the identified entity vectors into the text vector sequence of the legal document sample; Determine the maximum input length based on input requirements; Add special tags to the fused input vector sequence to help the model understand the text structure and task requirements; The goal setting of fine-tuning training includes the goal of generating legal document format, the goal of accuracy in the use of legal terminology, and the goal of constructing legal logical relationships; The legal document samples that integrate legal knowledge graph data and adjust the format are input into the model in batches according to the set size; After the data is input into the model, the pre-trained part is first encoded. Through the self-attention mechanism of the multi-layer Transformer encoder, the model can fully capture the semantic features of the text; The encoded semantic features will be passed to the generation layer for generating specific document types and the format judgment layer for judging the correctness of the document format for further processing and analysis; According to the model output and the real legal document samples, the loss value is calculated according to the set loss function. The loss function includes the basic language generation loss, the legal term loss penalty item, and the legal logic relationship loss penalty item. Specifically, it is: Based on language generation loss: For the language part generated by legal documents, the cross entropy loss function is used; let the word probability distribution predicted by the model be p(yx), p mnk represents the probability of predicting the kth word in the vocabulary for the mth sample and the nth word position, y mnk The mth sample and the nth word position predict the true label of the kth word in the vocabulary. The batch size is B, the sample length is S, and the vocabulary size is V. The cross entropy loss formula for the language generation part is: Where k is the index of the vocabulary dimension; Legal term loss penalty: set as the weight coefficient ω of legal term loss term To emphasize the correct use of legal terms, a penalty term is added; define the function f tern (x) Calculate the number of legal term errors in the mth sample; assuming that there are a total of T legal terms to be checked, the legal term loss penalty formula is: Legal logic relationship loss penalty item: For the legal logic relationship part, a penalty item is also added; set as the weight coefficient of the legal logic relationship loss ω logic , define the function f logic (x) Calculate the degree of legal logic relationship error in the mth sample; assuming that there are a total of R legal logic relationships to be checked, the legal logic relationship loss penalty formula is: Total loss function: L2 = L lang +L term +L logic ; The back-propagation algorithm is used to calculate the gradient of the loss to the model parameters. Starting from the last layer, the gradient value of each layer parameter is calculated forward in sequence through the chain rule to obtain the gradient vector corresponding to each parameter; the Adam optimizer is used to update the model parameters.

6. According to claim 5, a method for intelligently generating legal documents based on a large model is characterized in that: In step 5, the user selects the document type and inputs relevant parameters through the friendly interactive interface provided by the system, and the interface transmits the information to the fine-tuned large model; The model determines the draft framework based on the pre-learned structural template and legal logic of this type of document, and generates a draft of the legal document based on legal knowledge and language skills.

7. The method for intelligently generating legal documents based on a large model according to claim 6 is characterized in that: In step 6, the quality of the first draft of the legal document is evaluated from the dimensions of language fluency, legal terminology accuracy, logical rationality, and format standardization; According to the evaluation results, the draft of the legal document that needs to be optimized and its corresponding evaluation data are collected and marked as training data, and input into the model for fine-tuning training; The legal documents generated after optimization and model retraining are evaluated again. When the quality meets the standards, they are output in the format required by the user, and the generation and optimization process data are recorded.

8. The intelligent generation system of legal documents based on a large model according to claim 7 is applied to the intelligent generation method of legal documents based on a large model according to claims 1-7, characterized in that: It includes data collection and preprocessing module, legal knowledge graph construction module, model pre-training module, model fine-tuning training module, legal document generation module, and evaluation and optimization module; The data collection and preprocessing module collects legal-related texts through multiple channels, and cleans, normalizes, and classifies and annotates the collected data; The legal knowledge graph construction module performs knowledge extraction, knowledge fusion and disambiguation operations on legal texts, and integrates the processed information to obtain a legal field knowledge graph; The model pre-training module performs a pre-training operation on the selected BERT-large architecture model; The model fine-tuning training module fine-tunes the model for the legal document generation task; The legal document generation module receives the document type and related parameter information input by the user and passes them to the model, and generates a first draft of the legal document based on the document structure template and legal logic learned by the model; The evaluation and optimization module evaluates the quality of the first draft of the legal document in multiple dimensions, optimizes the model based on the evaluation results and outputs qualified documents, while recording the generation and optimization process data.

Images