A large model-based legal document intelligent generation method and system

Abstract

The application discloses a kind of legal document intelligent generation method and system based on large model, it is related to natural language processing technical field.The method is by legal relevant text and pre-processing through multi-channel acquisition, constructs legal domain knowledge graph, with BERT model pre-training and graph and document sample are fused and fine-tuning training;User selects document type and input parameter with the aid of interactive interface and transmits to model generation legal document first draft, again by multidimensional evaluation and optimization cycle, until the legal document of output standard is reached;The method covers data processing, knowledge graph construction, model training and document generation evaluation whole process, utilizes multi-source data and enriches knowledge reserve, and is combined with pre-training and fine-tuning to improve the pertinence of model, multidimensional evaluation guarantees document quality.Effectively solve legal document writing efficiency, quality is unstable and other problems, improve legal document generation efficiency and quality, promote the intelligent development of legal field.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and more specifically, to a method and system for intelligent generation of legal documents based on a large model. Background Technology

[0002] In recent years, artificial intelligence (AI) technology has flourished, and deep learning technology has made significant progress, greatly enhancing natural language processing (NLP) capabilities. This has not only improved the performance of traditional NLP tasks such as machine translation and text classification, but also created opportunities for emerging application areas such as automated document generation. Particularly in the legal industry, where there are stringent requirements for precise language expression and standardized text formats, there is an urgent need for technology capable of automatically generating legal documents that meet specific needs.

[0003] Currently, some tools on the market attempt to automate the production of legal documents through rule engines or simple template filling mechanisms. For example, some basic products allow users to input specific case details based on a pre-defined document structure, thereby generating preliminary legal documents. However, such methods lack flexibility and adaptability, making it difficult to cope with complex and ever-changing real-world business needs. Furthermore, although some researchers have used shallow neural network models for text generation, which has improved the quality of generated text to some extent, its performance in terms of the coherence and logical consistency of long texts is unsatisfactory.

[0004] Overall, existing automated legal document generation tools either rely too heavily on fixed rule bases and lack the ability to handle novel situations, or are based on basic artificial intelligence algorithms and cannot guarantee that the generated documents have a professional level and accuracy. These shortcomings limit their widespread application in actual workflows. Summary of the Invention

[0005] In view of the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for intelligent generation of legal documents based on large models.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] Firstly, a method for intelligent generation of legal documents based on a large model includes the following steps:

[0008] Step 1: Collect legal-related texts through multiple channels and preprocess the collected data;

[0009] Step 2: Perform knowledge extraction, knowledge fusion, and disambiguation operations on the preprocessed legal-related texts, and integrate the processed information to obtain a legal knowledge graph;

[0010] Step 3: Selecting the BERT model for pre-training, including model initialization, configuring hyperparameters, determining pre-training tasks, data preparation, and the pre-training process;

[0011] Step 4: Fusing the legal domain knowledge graph data with the legal document sample data and adjusting the input format, setting the fine-tuning training target, and inputting the model in batches for fine-tuning training;

[0012] Step 5: The user selects the document type and inputs the relevant parameters through the interactive interface, which are passed to the fine-tuned model. The model determines the framework of the preliminary draft based on the document type and generates the preliminary draft of the legal document;

[0013] Step 6: Multi-dimensional evaluation of the preliminary draft quality. If optimization is needed, the evaluation results are fed back to the model training link for targeted improvement. After meeting the requirements, the legal document that meets the requirements is output to the user.

[0014] Preferably, in step 2, the knowledge extraction includes entity extraction, relation extraction, and attribute extraction; the knowledge fusion and disambiguation include entity alignment, entity disambiguation, and knowledge integration.

[0015] Preferably, in step 3, the BERT-large architecture contains 24 layers of Transformer encoder, with a hidden layer size of 1024 and 16 self-attention heads. The learning rate is set between 1e-6 and 3e-6. The training batch size is commonly set to 16 or 32. Due to the depth and complexity of the BERT-large model, the number of training rounds is generally set to 10-20 rounds. The Adam optimizer is selected.

[0016] Design pre-training tasks, namely Masked Language Model (MLM) task and Next Sentence Prediction (NSP) task.

[0017] Collect large-scale general text data as pre-training materials and preprocess these data.

[0018] Input the preprocessed general text data into the pre-training model according to the set batch size. For each batch of data, the model performs forward propagation calculation according to the set pre-training task. Specifically:

[0019] For the MLM task, the cross-entropy loss function is used to calculate the cross-entropy loss between the predicted word probability distribution and the one-hot encoding of the true word. Assuming the vocabulary size is V, the number of masked words is N, and the predicted probability of the jth word in the vocabulary for the ith masked word is p ij , the true label is y ij , then the loss function formula of the MLM task is:

[0020]

[0021] wherein the real label y ij Only the position corresponding to the real word is 1, and the rest is 0;

[0022] For the NSP task, the cross-entropy loss between the predicted sentence relationship probability and the real label is also calculated using the cross-entropy loss function; assuming that the probability of predicting that two sentences are adjacent is p, and the real label is y, then the loss function of the NSP task is:

[0023] Loss NSP =-[ylog(p)+(1-y)log(1-p)]

[0024] wherein the real label y is 1 when it means adjacent, and 0 when it means not adjacent;

[0025] The total loss Loss1 = Loss MLM +Loss NSP is calculated.

[0026] The optimization algorithm Adam is used for back propagation to adjust the parameters of the model, so that the model is gradually optimized in the direction of reducing the loss value and better completing the pre-training task;

[0027] For the selected Adam optimizer, it will update the parameters according to the first moment estimate and second moment estimate of the gradient and the set learning rate; its update formula is:

[0028] m t =β1m t-1 +(1-β1)g t

[0029]

[0030]

[0031]

[0032] wherein wherein m t and v t are the values of the first moment estimate and the second moment estimate at the t-th step, g t is the gradient at the t-th step, α is the learning rate, β1, β2, ε are the hyperparameters of the optimizer, and are the corrected first moment estimate and second moment estimate, θ t is the value of the model parameter at the t-th step, θ t+1 is the updated parameter value;

[0033] According to the set number of training rounds, multiple batches of data are repeatedly iteratively trained.

[0034] Preferably, in step 4, key information including entities, relationships and attributes is extracted from the constructed legal domain knowledge graph, and the extracted entities and relationships are vectorized;

[0035] For each legal document sample, the text content therein is traversed to identify the words or phrases corresponding to the entities in the knowledge graph; the identified entity vectors are fused into the text vector sequence of the legal document sample;

[0036] According to the input requirements, the maximum input length is determined; special marks are added to the fused input vector sequence to assist the model in understanding the text structure and task requirements;

[0037] The fine-tuning training target setting includes a legal document format generation target, a legal terminology application accuracy target, and a legal logic relationship construction target;

[0038] The legal document samples fused with the legal domain knowledge graph data and adjusted in format are input into the model in turn according to the set batch size;

[0039] After the data is input into the model, it is first encoded by the pre-trained part. Through the self-attention mechanism of the multi-layer Transformer encoder, the model can fully capture the semantic features of the text;

[0040] The encoded semantic features are passed to the generation layer for generating a specific document type and the format judgment layer for judging the correctness of the document format for further processing and analysis;

[0041] According to the model output and the real legal document sample, the loss value is calculated according to the set loss function, which includes a basic language generation loss, a legal terminology loss penalty term, and a legal logic relationship loss penalty term, specifically:

[0042] Based on the language generation loss: for the language part of the legal document generation, the cross-entropy loss function is used; let the model predict the word probability distribution as p(y|x), p mnk represents the probability of predicting the kth word in the vocabulary table for the mth sample, the nth word position, y mnk The real label for the kth word in the vocabulary table for the mth sample, the nth word position, the batch size is B, the sample length is S, and the vocabulary size is V. The cross-entropy loss formula of the language generation part is:

[0043]

[0044] Where k is the index of the vocabulary dimension;

[0045] Legal terminology loss penalty term: let ω termTo emphasize the correct use of legal terms, a penalty term is added; define the function f tern (x) Calculate the number of legal term errors in the mth sample; assuming a total of T legal terms to be checked, the legal term loss penalty term formula is:

[0046]

[0047] Legal logic relationship loss penalty term: For the legal logic relationship part, a penalty term is also added; set the weight coefficient ω of legal logic relationship loss logic , define the function f logic (x) Calculate the degree of legal logic relationship error in the mth sample; assuming a total of R legal logic relationships to be checked, the legal logic relationship loss penalty term formula is:

[0048]

[0049] Total loss function: L2 = L lang +L term +L logic ;

[0050] The loss is calculated using the back propagation algorithm to calculate the gradient of the model parameters, starting from the last layer, and then calculating the gradient value of each layer parameter in turn, to get the gradient vector corresponding to each parameter; the Adam optimizer is used to update the parameters of the model.

[0051] Preferably, in step 5, the user selects the type of document and inputs the relevant parameters through the friendly interface provided by the system, and the interface transmits the information to the fine-tuned large model;

[0052] The model determines the framework of the preliminary draft according to the structure template and legal logic of the type of document learned in advance, and generates the preliminary draft of the legal document combining legal knowledge and language ability.

[0053] Preferably, in step 6, the quality of the preliminary draft of the legal document is evaluated from the dimensions of language fluency, legal term accuracy, logical reasonableness, and format specification;

[0054] According to the evaluation results, collect the legal document preliminary draft that needs to be optimized and its corresponding evaluation data and mark it as training data, and input the model for fine-tuning training;

[0055] The legal document generated after optimization and model retraining is evaluated again, and when the quality meets the standard, it is output in the format required by the user, and the generation and optimization process data are recorded.

[0056] In a second aspect, the application discloses a large model-based intelligent legal document generation system, which comprises a data collection and preprocessing module, a legal knowledge graph construction module, a model pre-training module, a model fine-tuning training module, a legal document generation module and an evaluation and optimization module.

[0057] The data collection and preprocessing module collects legal related texts through multiple channels and performs cleaning, standardization and classification annotation on the collected data.

[0058] The legal knowledge graph construction module performs knowledge extraction, fusion and disambiguation on legal texts, and integrates the processed information to obtain a legal domain knowledge graph.

[0059] The model pre-training module pre-trains a selected BERT-large architecture model.

[0060] The model fine-tuning training module fine-tunes the model for a legal document generation task.

[0061] The legal document generation module receives a document type and related parameter information input by a user and transmits the information to the model, and generates a legal document draft according to a document structure template and legal logic learned by the model.

[0062] The evaluation and optimization module evaluates the quality of the legal document draft in multiple dimensions, optimizes the model according to the evaluation result and outputs a qualified document, and records data of the generation and optimization process.

[0063] Compared with the prior art, the application has the following beneficial effects:

[0064] 1. In the application, legal related texts are collected through multiple channels, so that legal data of different sources and different fields can be widely covered, and the richness and comprehensiveness of the data are ensured. From an official legal database to a professional legal document library, a well-known legal academic resource platform and a file published by a government department, the multi-source data become standardized and orderly after preprocessing, and high-quality materials are provided for subsequent links, so that the tediousness of manual collection and arrangement is avoided, a large amount of time and effort is saved, and the overall work efficiency is improved.

[0065] 2. In the application, the legal domain knowledge graph constructed by using knowledge extraction, fusion and disambiguation operations integrates legal knowledge scattered in various places and structures complex legal relationships, entities and other information. In the generation of a document, the model can quickly and accurately obtain the required knowledge, so that the content of the document is more accurate and professional, the quality of the document is improved, and problems caused by lack of knowledge or incorrect reference are reduced.

[0066] 3、In the application, by selecting the BERT-large architecture for pre-training, its powerful language processing capability combined with carefully designed pre-training tasks (such as MLM and NSP tasks) enables the model to well grasp the semantics, syntax and logical relationship of language, laying a foundation for subsequent integration of legal knowledge. In the fine-tuning training stage, the legal document format generation, legal terminology application accuracy and legal logical relationship construction are set as the goals, and the loss function containing the corresponding penalty term is used to prompt the model to focus on learning and accurately applying legal professional knowledge, ensuring that the generated document is normative in format, accurate in terminology application and rigorous in logic, showing a high degree of professionalism.

[0067] 4、In the application, the user only needs to operate simply on the friendly interaction interface, select the document type and input the key parameters, and the system can rely on the pre-trained and fine-tuned large model to quickly determine the framework of the preliminary draft and generate the preliminary legal document. Compared with traditional manual writing, the document generation time is greatly shortened, especially for some common legal documents with relatively fixed format and regular content, the efficiency is improved more significantly.

[0068] 5、In the application, through the multi-dimensional quality evaluation mechanism, the language fluency, legal terminology accuracy, logical rationality and format standardization of the preliminary draft can be comprehensively and meticulously checked, problems can be found in time and relevant data can be fed back to the model for targeted fine-tuning training, and after multiple iterations, the quality of the document is continuously optimized to ensure that the final output document meets the high standard legal requirements, effectively reducing the possibility of subsequent legal risks caused by poor quality documents. BRIEF DESCRIPTION OF DRAWINGS

[0069] Figure 1 A flowchart of a legal document intelligent generation method based on a large model is proposed for the application;

[0070] Figure 2 A flowchart of a legal document intelligent generation system based on a large model is proposed for the application. DETAILED DESCRIPTION

[0071] Embodiments further illustrate the legal document intelligent generation method and system based on a large model proposed by the application.

[0072] REFERENCE Figure 1 A legal document intelligent generation method based on a large model, comprising the following steps:

[0073] Step 1: Collect legal related texts from multiple channels, and preprocess the collected data;

[0074] Step 2: Perform knowledge extraction, knowledge fusion and disambiguation operations on the preprocessed legal related texts, and integrate the processed information to obtain a legal domain knowledge graph;

[0075] Step 3: Selecting the BERT model for pre-training, including model initialization, configuring hyperparameters, determining pre-training tasks, data preparation, and pre-training process;

[0076] Step 4: Fusing legal domain knowledge graph data with legal document sample data and adjusting input format, setting fine-tuning training target, inputting model in batches for fine-tuning training;

[0077] Step 5: User selects document type and inputs related parameters through interactive interface, which are passed to the fine-tuned model. The model determines the framework of the preliminary draft according to the document type and generates the preliminary draft of the legal document;

[0078] Step 6: Multi-dimensional evaluation of the preliminary draft quality. If optimization is needed, the evaluation results are fed back to the model training link for targeted improvement. After meeting the requirements, the legal document that meets the requirements is output to the user.

[0079] In step 1, legal-related texts are collected from official legal databases, professional legal document libraries, well-known academic resources platforms, and government-issued legal regulations and policy document originals. Data preprocessing includes data cleaning, text normalization, and classification labeling.

[0080] Data cleaning: removes duplicate content in the text to avoid interference with model training by redundant data; identifies and deletes text fragments with format errors, garbled characters, and obvious incompleteness to improve data quality and usability.

[0081] Text normalization: unifies the encoding format of the text to the universal UTF-8 for easy processing and storage; standardizes the use of punctuation marks, such as converting full-width punctuation to half-width punctuation, and correcting punctuation abnormalities caused by input errors or non-standardization; corrects common misspellings and grammatical errors to make the text more consistent with language standards and facilitate subsequent natural language processing operations.

[0082] Classification labeling: classifies texts according to legal domain categories (such as civil law, criminal law, administrative law, etc.) for subsequent analysis and processing according to the characteristics of different domains; labels key elements in legal texts, such as legal subjects (party names, corporate names, etc.), legal acts (contract signing, litigation, torts, etc.), legal relationships (contractual rights and obligations, causation in tort cases, etc.), and cited legal clause numbers or names, etc., to provide more accurate information for knowledge extraction and model training.

[0083] In step 2, knowledge extraction includes entity extraction, relation extraction, and attribute extraction; knowledge fusion and disambiguation include entity alignment, entity disambiguation, and knowledge integration;

[0084] Entity Extraction: Utilizing named entity recognition technology, combined with legal dictionaries and grammatical rules, this technology accurately identifies various entities from legal texts. Besides common names of people, places, and organizations, it focuses on identifying specific legal entities, such as legal provision numbers, legal procedure names (filing, appeal, retrial, etc.), types of legal liability (breach of contract, tort liability, etc.), and types of legal documents (contracts, wills, judgments, etc.). For example, in a contract text, it identifies "Party A" and "Party B" as contracting parties, "Contract Signing Date" as a time entity, and "Subject Matter" as a specific contract-related entity.

[0085] Relation extraction: Based on dependency parsing and semantic role labeling techniques, relationships between entities are determined by analyzing sentence structure and semantics. For example, in a litigation case text, the "adversarial" relationship between "plaintiff" and "defendant," the "trial" relationship between "court" and "case," and the "inclusion" relationship between "contract" and "clause" can be determined. For some complex legal relationships, such as the citation relationships between legal provisions or the hierarchical relationships between legal concepts, specialized legal knowledge analysis tools or manual annotation can be used to determine them.

[0086] Attribute extraction: For the identified entities, extract their relevant attribute information; for example, for the "natural person" entity, extract attributes such as age, gender, occupation, and ID number; for the "contract" entity, extract attributes such as contract amount, contract term, and place of performance; for the "legal provision" entity, extract attributes such as the promulgation time, number of revisions, and scope of application of the provision to further enrich the descriptive information of the entity.

[0087] Entity alignment: Since different data sources may have multiple ways of representing the same entity, such as "People's Republic of China" being abbreviated as "China" and "Limited Liability Company" being abbreviated as "Limited Company", etc., by calculating entity name similarity, context-based semantic similarity, and using external authoritative knowledge bases (such as business registration information, government agency directories, etc.) for verification, these entities that refer to the same entity are merged and aligned to ensure that each entity in the knowledge graph has a unique identifier.

[0088] Entity disambiguation: For entities with the same name but different actual entities, such as companies with the same name in different regions or legal concepts with the same name in different legal contexts, the entity's attribute information, contextual information, and relevant domain knowledge are used to distinguish and disambiguate them. For example, in a commercial dispute involving companies in multiple regions, the specific "XX Company" entity is determined by the company's registered address, business scope, and other attributes, thus avoiding entity confusion in the knowledge graph.

[0089] Knowledge integration: integrate the extracted, fused and disambiguated entity, relationship and attribute information to construct a structured legal domain knowledge graph; use a graph database (such as Neo4j) for storage, treat entities as nodes in the graph, treat relationships as edges connecting nodes, and treat attributes as additional information of nodes, to intuitively display the network structure and logical relationship between legal knowledge; for example, in the knowledge graph, the "sales contract" node is connected to the "buyer" and "seller" nodes through the "contract party" edge, and the "buyer" node displays its specific information through the "name", "address" and other attribute edges, in this way, the internal relationship of legal knowledge is clearly presented, which provides strong support for subsequent models to use knowledge graph for reasoning and generating legal documents.

[0090] In step 3, the BERT-large architecture contains 24 layers of Transformer encoder, the hidden layer size is 1024, the number of self-attention heads is 16, the learning rate is set between 1e-6 and 3e-6, and the learning rate is adjusted by using the linear decay method; due to the depth and complexity of the BERT-large model, it is generally set to start at about 10-20 rounds; the Adam optimizer is selected.

[0091] Design pre-training tasks, i.e. Masked Language Model (MLM) task and Next Sentence Prediction (NSP) task; when performing Masked Language Model (MLM) task, accurately extract the output vector corresponding to the masked position, convert it to a probability distribution on the vocabulary table through a classification layer, and use it to predict the masked word; for the Next Sentence Prediction (NSP) task, accurately extract the output vector corresponding to the [CLS] mark, and get the binary classification probability distribution of sentence relationship through a linear layer and an activation function to determine whether the two sentences are adjacent.

[0092] Collect large-scale general text data as pre-training materials, and preprocess these data;

[0093] The general text data has a wide range of sources, which can include news articles, blogs, novels, academic papers and other types of text to ensure that the model can learn a variety of language expressions and semantic information; for example, text data can be collected from news websites, literature websites, academic databases and other platforms on the Internet, and the data volume usually reaches tens of billions or even hundreds of billions of words;

[0094] The preprocessing includes data cleaning, word segmentation and conversion to the input format required by the model;

[0095] Data cleaning: remove noise data in text, such as HTML tags, special characters (such as non-printable characters, garbled characters, etc.), duplicate data, etc.; for example, use regular expression to match and delete HTML tags, filter out control characters and garbled characters in text, and remove duplicate text paragraphs or sentences through data duplication checking algorithm;

[0096] Word segmentation: divide text into words or word blocks; for English text, common English word segmentation tools can be used, such as word segmentation functions in NLTK library, to divide text into word sequences; for Chinese text, Chinese word segmentation tools can be used, such as Jieba word segmentation, to divide text into word sequences;

[0097] Convert to input format required by model: according to the requirements of BERT model, mark the segmented text and add special marks such as [CLS] (representing the beginning of the text) and [SEP] (representing the end of the sentence); at the same time, convert the text into a tensor format that the model can process, for example, map words or word blocks to corresponding index sequences, and truncate or pad the text according to the set maximum input length to ensure that the length of all input texts is consistent.

[0098] Input the preprocessed general text data into the pre-trained model according to the set batch size, and for each batch of data, the model performs forward propagation calculation according to the set pre-training task, which is:

[0099] For MLM task, use cross-entropy loss function to calculate the cross-entropy loss between the probability distribution of predicted word and the one-hot encoding of real word; assuming the size of vocabulary is V, the number of masked words is N, the predicted probability of the i-th masked word at the j-th word in the vocabulary is p ij , the real label is y ij , then the loss function formula of MLM task is:

[0100]

[0101] Where, the real label y ij is 1 only for the position corresponding to the real word, and 0 for the rest;

[0102] For NSP task, also use cross-entropy loss function to calculate the cross-entropy loss between predicted sentence relationship probability and real label; assuming the probability of predicting that two sentences are adjacent is p, and the real label is y, then the loss function of NSP task is:

[0103] Loss NSP =-[ylog(p)+(1-y)log(1-p)]

[0104] Wherein, the real label y is 1 when indicating adjacent, and 0 when indicating not adjacent;

[0105] Calculate the total loss Loss1 = Loss MLM +Loss NSP .

[0106] The optimization algorithm Adam is used for back propagation to adjust the parameters of the model, so that the model is gradually optimized in the direction of reducing the loss value and better completing the pre-training task;

[0107] For the selected Adam optimizer, it will update the parameters according to the first-order moment estimate and second-order moment estimate of the gradient and the set learning rate; its update formula is:

[0108] m t = beta1m t-1 + (1-beta1)g t

[0109]

[0110]

[0111]

[0112] Wherein, wherein m t and v t are the values of the first-order moment estimate and the second-order moment estimate at the t-th step, g t is the gradient at the t-th step, alpha is the learning rate, beta1, beta2, epsilon are the hyperparameters of the optimizer, and are the corrected first-order moment estimate and second-order moment estimate, theta t is the value of the model parameter at the t-th step, theta t+1 is the updated parameter value.

[0113] According to the set training round number, a plurality of batches of data are repeatedly iteratively trained.

[0114] In step 4, key information including entities, relationships and attributes is extracted from the constructed legal field knowledge graph, and the extracted entities and relationships are represented by vectors;

[0115] For example, for a contract-related knowledge graph, the contract parties (entities), contract signing relationship (relationship), and registered address and contact information of the parties (attributes) are extracted; the extracted entities and relationships are converted into vector form using a word vector representation method (such as Word2Vec, GloVe, or a word vector generated based on a pre-trained BERT model); assuming that the entity "Party A" is vectorized to obtain the vector e 甲方, the relationship "contract signing" is converted into a vector r 合同签订 .

[0116] For each legal document sample, traverse the text content therein, identify the words or phrases corresponding to the entities in the knowledge graph; fuse the identified entity vectors into the text vector sequence of the legal document sample;

[0117] For example, in a contract document, when the entity "Party A" is identified as matching the corresponding entity in the knowledge graph, the vector e 甲方 of the "Party A" entity is fused into the vector sequence near the position where "Party A" appears in the text; direct concatenation, weighted summation, etc. fusion strategy can be used; if direct concatenation is used, let the text vector sequence of the legal document sample at a certain position be s = [v1, v2, …, v n ], when the "Party A" entity is identified to correspond to v i , the fused vector sequence becomes s' = [v1, v2, …, v i-1 , e 甲方 , v i , …, v n ].

[0118] According to the input requirements, determine the maximum input length; add special marks to the fused input vector sequence to assist the model in understanding the text structure and task requirements;

[0119] According to the model input requirements and the characteristics of hardware resources and training data, determine the maximum input length L, for example, set it to 512 vector unit lengths (the specific value can be determined according to actual experiments and adjustments); Add special marks to the fused input vector sequence, add [CLS] marks at the beginning to represent the global features of the entire text, and add [SEP] marks between different sentences or paragraphs to distinguish semantic units, helping the model better understand the text structure and task requirements; For example, the fused vector sequence finally becomes s final = [[CLS], e 甲方 , v1, v2, …, [SEP], v i+1 , …, v n ], if the vector sequence length is less than L, perform padding operation (such as filling with zero vector) to make its length reach L.

[0120] The fine-tuning training target setting includes legal document format generation target, legal terminology application accuracy target, and legal logic relationship construction target;

[0121] Legal document format generation target: Develop detailed format templates for different types of legal documents (such as contracts, complaints, judgments, etc.), requiring the model to accurately generate fixed format content for the beginning part of the corresponding document (such as the title, number, signing date, party information of the contract, the name of the document, the basic information of the plaintiff and the defendant of the complaint, etc.), the main part reasonably arranges the clauses or statements (such as the specific clause settings of the contract, the facts and reasons of the complaint, etc.), and the end part meets the standard requirements (such as the signature and seal column of the contract, the inscription of the complaint, etc.);

[0122] Legal terminology accuracy target: Build a comprehensive legal terminology database covering various legal fields commonly used professional terms and their accurate interpretation, usage examples, etc.; by setting appropriate weight coefficients ω term (For example, ω term = 0.5), add a legal terminology loss penalty term to the loss function, when the model generates text with legal terminology errors (such as misusing "deposit" as "subscription", etc.), this term will increase the loss value, prompting the model to pay attention to and accurately use legal terminology;

[0123] Legal logic relationship construction target: Develop a series of legal logic rules (such as the rule of equal rights and obligations in contracts, the reasonable derivation rule between litigation requests and factual and legal bases in litigation documents, etc.), add a legal logic relationship loss penalty term to the loss function by setting the corresponding weight coefficient ω logic (For example, ω logic = 0.3), when the model generates text with logical relationship errors (such as mismatched rights and obligations, lack of reasonable basis for litigation requests, etc.), the model will be guided to construct content that conforms to legal logic through this penalty.

[0124] The legal document samples that have been fused with legal domain knowledge graph data and adjusted in format are input into the model in batches according to the set batch size (such as 16 or 32, etc., according to hardware resources and data volume to choose reasonably).

[0125] After the data is input into the model, it is first encoded by the pre-trained part, and through the self-attention mechanism of the multi-layer Transformer encoder, the model can fully capture the semantic features of the text.

[0126] The encoded semantic features will be passed to the generation layer for generating specific document types and the format judgment layer for judging the correctness of the document format for further processing and analysis.

[0127] According to the model output and the real legal document samples, the loss value is calculated according to the set loss function, which includes basic language generation loss, legal terminology loss penalty term and legal logic relationship loss penalty term, specifically:

[0128] Language generation loss: for the language part of the legal document generation, the cross-entropy loss function is used; let the model's predicted word probability distribution be p(y|x), p mnk represents the probability of predicting the k-th word in the vocabulary at the n-th word position of the m-th sample, y mnk The true label for the m-th sample, the n-th word position, and the k-th word in the vocabulary is y

[0129]

[0130] where k is the index of the vocabulary dimension;

[0131] Legal term loss penalty term: let ω term Add a penalty term to emphasize the correct use of legal terms; define the function f tern (x) Calculate the number of legal term errors in the m-th sample (determined by comparing with the correct legal term label); assume that there are T legal terms to be checked in total, then the legal term loss penalty term formula is:

[0132]

[0133] Legal logic relationship loss penalty term: for the legal logic relationship part, also add a penalty term; let ω logic be the weight coefficient of the legal logic relationship loss, and define the function f logic (x) Calculate the degree of legal logic relationship error in the m-th sample (determined by rule checking, logic template matching, etc.); assume that there are R legal logic relationships to be checked in total, then the legal logic relationship loss penalty term formula is:

[0134]

[0135] Total loss function: L2 = L lang + L term + L logic .

[0136] Use the backpropagation algorithm to calculate the gradient of the loss with respect to the model parameters. Starting from the last layer, calculate the gradient value of each layer's parameters in turn, and obtain the complete gradient vector corresponding to each parameter; for example, for a certain layer parameter θ in the model, its gradient calculation process is as follows: first calculate the gradient of the total loss L2 with respect to the output of the last layer where o is the output of the last layer), then calculate the gradient of the weight matrix with respect to the weight matrix W according to the activation function and weight matrix of this layer and the gradient of the bias vector b In this way, the gradient of each layer parameter is calculated in turn from the last layer to the front.

[0137] The model parameters are updated according to the gradient vector and the set learning rate and other parameters using the Adam optimizer. According to the update formula of the Adam optimizer (as described in the pre-training part), the model parameters are updated according to the first moment estimate m t and the second moment estimate v t and the learning rate a Through multiple rounds of fine-tuning training, the performance of the model on the legal document generation task is gradually improved, enabling it to generate high-quality legal documents that meet legal norms and user needs.

[0138] In step 5, the user selects the document type and inputs the relevant parameters through the friendly interface provided by the system, and the interface transmits the information to the fine-tuned large model;

[0139] The model determines the framework of the preliminary draft based on the pre-learned structure template and legal logic of this type of document, and generates a preliminary draft of the legal document based on legal knowledge and language ability.

[0140] In step 6, the quality of the preliminary draft of the legal document is evaluated from the dimensions of language fluency, accuracy of legal terminology, logical rationality, and format standardization;

[0141] Language fluency: the perplexity index of the text is calculated using the language model in natural language processing technology to evaluate language fluency; let the text be T=[w1,w2,…,w n ], where w i represents the i-th word, and the language model calculates the word probability distribution as P(w i w 1；i-1 ), then the calculation formula of perplexity is: The lower the perplexity, the more natural and fluent the text is in language expression, and it conforms to normal language habits and language rules;

[0142] Legal terminology accuracy: compare all the words in the preliminary draft with the pre-constructed legal terminology library; let E be the set of legal terms in the preliminary draft, C be the set of correct legal terms, f term (e) be a function that judges whether the term e is correct (if correct, f term (e)=0, and if incorrect, f term (e)=1), then the calculation formula of the number of legal terminology errors N term is:

[0143] Logical rationality: logical rules checker and legal reasoning model are combined to evaluate logical rationality; let R be the set of logical relations in the text, f logic (r) is a function of judging whether the logical relation r is reasonable (f logic (r) = 0 if reasonable, f logic (r) = 1 if unreasonable), then the calculation formula of the number of legal logical relation errors N logic is:

[0144] Format specification: according to the standard format template of different types of legal documents, the matching check is carried out; let F be the set of format elements in the draft, C f be the set of format elements in the correct format template, f format (f) is a function of judging whether the format element f is correct (f format (f) = 0 if correct, f format (f) = 1 if incorrect), then the calculation formula of the number of format errors N format is:

[0145] According to the evaluation results of each dimension, a comprehensive quality score is generated for the draft by using weighted average method; the weights of language fluency, legal terminology accuracy, logical rationality and format specification are respectively w lang = 0.2, w term = 0.3, w logic = 0.3, w format = 0.2 (the weights can be adjusted according to actual needs and priorities), after calculating the corresponding scores according to the evaluation indexes of each dimension, the comprehensive quality score S is obtained by weighted summation according to the weights; assuming that the language fluency score S lang , the legal terminology accuracy score S lang , the logical rationality score S lang , the format specification score S lang is obtained by the respective evaluation methods, then the comprehensive quality score calculation formula is: S = w lang S lang + w term S term + w logic S logic + w format S format .

[0146] According to the evaluation results, collect the legal document draft that needs to be optimized and its corresponding evaluation data and mark it as training data, and input the model for fine-tuning training.

[0147] The legal document generated after optimization and model retraining is evaluated again, and when the quality meets the standard, it is output in the required format of the user, and the generation and optimization process data are recorded.

[0148] Referring to Figure 2 A large model-based legal document intelligent generation system, comprising a data acquisition and preprocessing module, a legal knowledge graph construction module, a model pre-training module, a model fine-tuning training module, a legal document generation module, and an evaluation and optimization module.

[0149] The data acquisition and preprocessing module collects legal related texts from multiple channels, and performs cleaning, standardization and classification annotation processing on the collected data.

[0150] The legal knowledge graph construction module performs knowledge extraction, knowledge fusion and disambiguation operations on legal texts, and integrates the processed information to obtain a legal domain knowledge graph.

[0151] The model pre-training module pre-trains the selected BERT-large architecture model.

[0152] The model fine-tuning training module fine-tunes the model for legal document generation tasks.

[0153] The legal document generation module receives the document type and related parameter information input by the user and passes it to the model, and generates a legal document draft according to the document structure template and legal logic learned by the model.

[0154] The evaluation and optimization module evaluates the quality of the legal document draft in multiple dimensions, optimizes the model according to the evaluation results, and outputs the qualified document, while recording the generation and optimization process data.

[0155] The above is only the preferred embodiment of the present application, the protection scope of the present application is not limited to the above-mentioned examples, all technical solutions under the idea of the present application belong to the protection scope of the present application. It should be noted that for ordinary skilled in the art, some improvements and decorations without departing from the principles of the present application, these improvements and decorations should also be considered as the protection scope of the present application.

Claims

1. A large model-based legal document intelligent generation method, characterized in that, The method comprises the following steps: Step 1: Collecting legal related texts through multiple channels and preprocessing the collected data; Step 2: Knowledge extraction, knowledge fusion and disambiguation are performed on the preprocessed legal related texts, and the processed information is integrated to obtain a legal domain knowledge graph; Step 3: Pre-training the BERT model, including model initialization, configuring hyperparameters, determining pre-training tasks, data preparation, and pre-training process; The BERT-large architecture contains 24 layers of Transformer encoder with a hidden size of 1024 and 16 self-attention heads; the learning rate is set to between 0.0001 and 0.00001 ; the training batch size is commonly set to 16 or 32; due to the depth and complexity of the BERT-large model, the number of training epochs is generally set to 10-20 epochs; Selecting the Adam optimizer; Designing pre-training tasks, namely Masked Language Model (MLM) task and Next Sentence Prediction (NSP) task; Collecting large-scale general text data as pre-training materials and preprocessing these data; The preprocessed general text data is input into the pre-training model according to the set batch size, and for each batch of data, the model performs forward propagation calculation according to the set pre-training task, specifically: For the MLM task, the cross-entropy loss between the probability distribution of its predicted word and the one-hot encoding of the real word is calculated using the cross-entropy loss function; assuming the size of the vocabulary is , the number of masked words is , the predicted probability of the th masked word in the vocabulary is the th word, the real label is , and the loss function formula of the MLM task is: ​ wherein the true label Only the position corresponding to the true word is 1, and the rest is 0; For the NSP task, the cross-entropy loss function is also used to calculate the cross-entropy loss between the predicted sentence relationship probability and the true label; assuming that the probability of predicting that the two sentences are adjacent is , and the true label is , then the loss function of the NSP task is: wherein the true label is is 1 if adjacent, and 0 if not adjacent; calculating the total loss ; Using the optimization algorithm Adam for back propagation to adjust the parameters of the model, so that the model is gradually optimized in the direction of reducing the loss value and better completing the pre-training task; for the selected Adam optimizer, it updates the parameters according to the first moment estimate and second moment estimate of the gradient and the set learning rate; its update formula is: where and are the first and second moment estimates at step , is the gradient at step , is the learning rate, , , are the hyperparameters of the optimizer, and are the corrected first and second moment estimates, is the value of the model parameters at step , is the updated parameter value; According to the set number of training rounds, the data of multiple batches is iteratively trained; Step 4: Fusion and format adjustment of legal domain knowledge graph data and legal document sample data, clear fine-tuning training target setting, input the model in batches for fine-tuning training; Extracting key information from the constructed legal domain knowledge graph, including entities, relationships and attributes, and vectorizing the extracted entities and relationships; For each legal document sample, traverse the text content to identify the entity corresponding to the knowledge graph; the identified entity vector is fused into the text vector sequence of the legal document sample; According to the input requirements, determine the maximum input length; add special markers to the fused input vector sequence to assist the model in understanding the text structure and task requirements; The fine-tuning training target setting includes the legal document format generation target, the legal terminology application accuracy target, and the legal logic relationship construction target; The legal document sample fused with the legal domain knowledge graph data and adjusted in format is input into the model according to the set batch size; After the data is input into the model, it is first encoded by the pre-trained part, and through the self-attention mechanism of the multi-layer Transformer encoder, the model can fully capture the semantic features of the text; The encoded semantic features are passed to the generation layer for generating a specific document type and the format judgment layer for judging the correctness of the document format for further processing and analysis; According to the model output and the real legal document sample, the loss value is calculated according to the set loss function, and the loss function includes the basic language generation loss, the legal terminology loss penalty term and the legal logic relationship loss penalty term, specifically: Language generation loss: For the language part of the legal document generation, the cross-entropy loss function is used; let the model's predicted word probability distribution be , , represents the probability of the th word in the th word position prediction vocabulary of the th sample, , the true label of the th word in the th word position prediction vocabulary of the th sample, the batch size is , the sample length is , and the vocabulary size is , then the cross-entropy loss formula of the language generation part is: wherein, is an index of a vocabulary dimension; Legal term loss penalty term: set to the weight coefficient of legal term loss To emphasize the correct use of legal terms, add a penalty term; define the function Calculate the number of legal term errors in the th sample; assume that there are a total of legal terms to be checked, then the legal term loss penalty term formula is: Penalty for Loss Due to Legal Logic Relationship: A penalty item is also added for the legal logic relationship component; it is set as the weighting coefficient for the loss due to the legal logic relationship. Define function Calculate the first The degree of legal logical relationship errors in each sample; assuming a total of If a legal logical relationship needs to be examined, the formula for the penalty term for loss related to that legal logical relationship is as follows: Total loss function: ; The gradient of the loss to the model parameters is calculated using the back propagation algorithm, starting from the last layer, and the gradient value of each layer parameter is calculated in turn forward through the chain rule to obtain the gradient vector corresponding to each parameter; the model parameters are updated using the Adam optimizer; Step 5: The user selects the type of document and inputs the relevant parameters through the interactive interface, which are passed to the fine-tuned model. The model determines the framework of the preliminary draft according to the type of document and generates the preliminary draft of the legal document. Step 6: The quality of the preliminary draft is evaluated in multiple dimensions. If optimization is needed, the evaluation results are fed back to the model training process for targeted improvement. Once the standard is met, the legal document that meets the requirements is output to the user.

2. The method of claim 1, wherein, In step 1, legal-related texts are collected from official legal databases, professional legal document libraries, well-known legal academic resource platforms, and original texts of laws, regulations, and policy documents released by government departments. Data preprocessing includes data cleaning, text normalization, and classification labeling.

3. The method of claim 2, wherein, In step 2, knowledge extraction includes entity extraction, relation extraction, and attribute extraction. Knowledge fusion and disambiguation include entity alignment, entity disambiguation, and knowledge integration.

4. The method of claim 3, wherein, In step 5, the user selects the type of document and inputs the relevant parameters through the friendly interactive interface provided by the system. The interface passes the information to the large model that has been fine-tuned; The model determines the framework of the preliminary draft based on the pre-learned structure template and legal logic of this type of document, and generates the preliminary draft of the legal document combining legal knowledge and language ability.

5. The large model-based legal document intelligent generation method according to claim 4, characterized in that, In step 6, the quality of the preliminary draft of the legal document is evaluated from the dimensions of language fluency, accuracy of legal terminology, logical reasonableness, and format specification; According to the evaluation results, collect the legal document preliminary drafts that need to be optimized and their corresponding evaluation data, and label them as training data, then input them into the model for fine-tuning training; The legal document generated after optimization and model retraining is evaluated again. When the quality meets the standard, it is output in the format required by the user, and the generation and optimization process data are recorded.

6. A large model-based intelligent legal document generation system applied to the large model-based intelligent legal document generation method of claims 1-5, characterized in that, It includes data collection and preprocessing module, legal knowledge graph construction module, model pre-training module, model fine-tuning training module, legal document generation module, and evaluation and optimization module; The data collection and preprocessing module collects legal-related texts from multiple channels and cleans, normalizes, and classifies the collected data; The legal knowledge graph construction module performs knowledge extraction and knowledge fusion and disambiguation operations on legal texts, and integrates the processed information to obtain a legal domain knowledge graph; The model pre-training module pre-trains the selected BERT-large architecture model; The model fine-tuning training module fine-tunes the model for legal document generation tasks; The legal document generation module receives the type of document and related parameter information input by the user and passes them to the model. It generates the preliminary draft of the legal document based on the document structure template and legal logic learned by the model; The evaluation and optimization module evaluates the quality of the preliminary draft of the legal document in multiple dimensions, optimizes the model according to the evaluation results, and outputs the standard document while recording the generation and optimization process data.

Images