Contract risk clause identification method based on natural language processing

Abstract

The invention discloses a contract risk clause identification method based on natural language processing, and the method comprises the steps: firstly extracting a clause set from a contract data set, and each clause is one piece of sample data; secondly, uniformly preprocessing the text in each piece of sample data to generate a sentence-level vector; then constructing positive and negative sample pairs based on each sentence vector in the sentence-level vectors; and then learning the set of the positive and negative sample pairs by using a comparative learning method, and constructing a classifier to output a clause recognition result. According to the method, the semantic understanding capability of complex sentence patterns in a contract text is remarkably improved, and the distinguishing capability of the model on terms with similar semantics but different categories is improved.

Description

Technical Field

[0001] The present invention relates to the field of text analysis, and particularly to a method for identifying contract risk clauses based on natural language processing. Background Art

[0002] With the continuous complexity of social and economic activities, contracts play a crucial role in various business activities. However, due to the complexity of the business environment and the variability of legal provisions, business contract risks always exist. Correctly understanding and identifying contract risk clauses is an important means to ensure the smooth progress of enterprise transactions and avoid economic losses. Especially in large-scale projects, finance, insurance and other fields, risk control has become an important part of contract management. Therefore, how to automatically identify potential risk clauses from a large number of contract texts has become a key issue in contract intelligent review and risk control. Currently, the identification of contract risk clauses mainly relies on manual review. However, this method is inefficient and costly, and it is difficult to meet the processing requirements of large-scale contract documents.

[0003] With the rapid development of natural language processing technology, semantic understanding based on deep learning has been widely applied in fields such as legal text mining, intelligent question answering, and public opinion analysis, providing a new technical path for the identification of contract risk clauses. However, in practical applications, the main challenge faced by the identification of contract risk clauses is how to accurately understand the deep semantics of the clauses and effectively distinguish the subtle differences between risk clauses and non-risk clauses. Therefore, the present invention proposes a method for identifying contract risk clauses based on natural language processing. This method integrates technologies such as sentence semantic modeling, attention mechanism, and contrast learning enhancement, and can effectively achieve deep semantic understanding of contracts and extraction of differential features, providing technical support for intelligent contract risk control. Summary of the Invention

[0004] In order to effectively achieve contract risk identification, this paper proposes a method for identifying contract risks based on natural language processing, which can effectively identify contract risk clauses.

[0005] The specific technical solutions adopted by the present invention are as follows:

[0006] Step (1) Data acquisition and division: Extract a clause set from the contract dataset , is the number of clauses. Each clause (i.e., sample) is represented as , , where represents the clause number, represents the text of the clause, represents the label of the clause, 0 means no risk, and 1 means there is risk.

[0007] Step (2) For each sample The text in is uniformly preprocessed as follows: First, the clauses are split into sentences, each clause is separated into several sentences by punctuation marks, and noise data such as meaningless characters, numbers, and HTML tags are removed. Then, based on the stop-word list, the stop words without practical meaning are removed. After that, the ambiguous or variant contract terms are normalized. Finally, the overly long sentences are truncated or window-slice segmented to meet the maximum input length limit of the model; after preprocessing, each sample is represented as , where represents the set of sentences, is the m-th sentence in , and m is the number of sentences contained in the clause

[0008] Step (3) Generate sentence-level vectors: For each clause after preprocessing , is input into the existing pre-trained large language model SBERT (Sentence-BERT) to obtain the vector representations of all sentences in the clause , ( ), and d is the dimension of the sentence vector.

[0009] Step (4) Calculate attention weights: For each sentence vector in , its attention weight is calculated.

[0010] Step (5) Generate clause-level vectors: According to the attention weights, the sentence vectors are weighted and summed to obtain the vector representation of the clause .

[0011] Step (6) Construct positive and negative samples: The vector representation of each clause is input into the Transformer model with Dropout twice to generate two different intermediate representations; the two intermediate vectors generated from the same clause are used as a positive sample pair, and the intermediate vectors generated from clauses of different risk categories are used as negative sample pairs.

[0012] Step (7) Use the contrastive learning method to learn the set of positive and negative sample pairs. The goal is to maximize the similarity between positive sample pairs and minimize the similarity between negative sample pairs; among them, the similarity metric is calculated using cosine similarity, and the contrastive learning loss function uses the improved NT-Xent (Normalized Temperature-scaled Cross Entropy Loss).

[0013] Step (8) Classifier construction: The vector representations of all clauses It is input into a multi-layer perceptron for training. The cross-entropy loss function is used to measure the difference between the predicted probability and the true label, serving as the optimization objective for the classification task.

[0014] Step (9): Combine the contrastive learning loss function and the classification loss function as the overall loss function, and use an optimizer for training; input the target clause into the trained model, and output the prediction scores corresponding to the two categories of "risk-free" and "risky"; after Softmax normalization, obtain the prediction probability for each category.

[0015] Compared with traditional classification methods, the beneficial effects of the present invention are as follows:

[0016] 1. The present invention uses the pre-trained model SBERT to efficiently extract the deep semantic features of contract sentences, significantly improving the semantic understanding ability of complex sentence patterns in contract texts.

[0017] 2. The present invention uses the attention mechanism to dynamically weight and fuse the vectors of each sentence within a clause, overcoming the problem of semantic dilution caused by long clauses and generating a more informative clause-level representation.

[0018] 3. The present invention realizes the aggregation of vectors of similar clauses and the separation of dissimilar clauses through contrastive learning, significantly improving the model's ability to distinguish clauses with similar semantics but different categories. Description of the Drawings

[0019] Figure 1 It is a flowchart of the contract risk clause recognition method based on natural language processing according to the present invention;

[0020] Figure 2 It is a flowchart of the execution of the attention mechanism. Detailed Embodiments

[0021] Dataset: The dataset used in this experiment is mainly the Contract Understanding Atticus Dataset (CUAD). CUAD is a dataset specifically for contract understanding, covering key clauses in various contract types, including 510 commercial contracts, more than 13,000 clauses, and the category of each clause is manually annotated by legal experts.

[0022] In order to make the objectives, technical solutions and advantages of the present invention more clear and understandable, the following combines the Figure 1 and the Figure 2 to elaborate in detail on the contract risk clause recognition method based on natural language processing provided by the present invention patent, including the following steps:

[0023] Step (1) Data acquisition and division: Parse and divide the text of all contracts in the contract dataset, and extract the contract clause set , Let be the number of clauses. Each clause (i.e., sample) is represented as , where represents the clause number, represents the text in the clause, represents the label of the clause, 0 means risk-free, and 1 means risky;

[0024] Step (2) Perform unified preprocessing on the text in each sample :

[0025] 2-1. Perform sentence splitting on the clauses, separate each clause into several sentences according to punctuation marks, and remove noise data such as meaningless characters, numbers, and HTML tags;

[0026] 2-2. Remove words with no practical meaning based on the Chinese stop word list, such as "de", "le", "and", etc.;

[0027] 2-3. Standardize contract terms with ambiguity or variants, for example, unify "Party A", "this company", "our party" to the same identifier, and standardize the expressions of time, amount, unit, etc.;

[0028] 2-4. Truncate or perform sliding window splitting on overly long sentences to meet the maximum input length limit of the model;

[0029] After preprocessing, each sample is represented as , where represents the sentence set, is the m-th sentence in , and m is the number of sentences contained in the clause

[0030] Step (3) Generate sentence-level vectors: For each clause , input into the pre-trained model SBERT to obtain the vector representations of all sentences in the clause ( ), where

[0031] is the dimension of the sentence vector; For each sentence vector in

[0032]

[0033] Among them represents the attention weight of the sentence vector and 、 are trainable parameter matrices is the bias term The function is used to convert a set of real numbers into a probability distribution The function compresses the input into the interval (-1, 1) and is used for feature non-linear transformation

[0034] Step (5) Generate the clause-level vector: According to the attention weight weighted sum of the sentence vectors to obtain the clause vector representation , and the formula is

[0035]

[0036] Step (6) Construct positive and negative samples: Represent the vector set generated by all clauses as , is the risk label is the total number of clauses

[0037] 6-1. Construct a Transformer contrastive learning model with an integrated Dropout mechanism (Dropout rate of 0.2). When the model propagates forward, a random Dropout mask is automatically generated to dynamically mask neurons

[0038] 6-2. Input the vector representation of each clause in into the contrastive learning model twice to generate two intermediate vectors. The formula is as follows

[0039]

[0040] Among them and are different Dropout masks obtained by inputting the same clause twice and are the two output intermediate vectors

[0041] 6-3. Represent the two intermediate vectors generated by the same clause vector as a pair of positive samples ;

[0042] 6-4. Represent the intermediate vectors generated by different category clause vectors as a pair of negative samples ;

[0043] Step (7) uses the contrastive learning method to learn the set of positive and negative sample pairs, with the goal of maximizing the similarity between positive sample pairs and minimizing the similarity between negative sample pairs:

[0044] 7-1. Calculate the similarity of clause vectors using cosine similarity and The similarity calculation formula is as follows:

[0045]

[0046] where represents the dot product operation of vectors, represents the Euclidean norm;

[0047] 7-2. The contrastive learning loss function uses the improved NT-Xent (Normalized Temperature-scaled Cross Entropy Loss), and its formula is as follows:

[0048]

[0049] where is the contrastive learning loss function, is the temperature hyperparameter. For the convenience of expression, and respectively represent the positive sample and negative sample of sample , is 's negative sample set, is the number of positive sample pairs; compared with the standard contrastive learning method, in constructing NT-Xent in the present invention, only clause pairs with different labels are selected as negative sample pairs, so as to avoid the interference of pseudo-negative sample pairs with similar semantics on model training; this strategy significantly enhances the discriminability of clause representations, improves the model's ability to learn semantic boundaries, is applicable to practical tasks with inter-class ambiguity and class imbalance problems, and can provide a more reliable basis for subsequent risk identification;

[0050] Step (8) Classifier construction: Input the vector representations of all clauses into the MLP for training. It contains two hidden layers, and the dimension of each hidden layer is 512, including the GELU function and the Dropout function; the cross-entropy loss function is used to measure the difference between the predicted probability and the true label, and is used as the optimization objective for the classification task. The calculation formula is as follows:

[0051]

[0052] where is the total number of clauses, is the class index (1 represents no risk, 2 represents risk), is the The true label (0 or 1) of a sample in the category is, the probability value predicted for the sample in the category is the cross-entropy loss function;

[0053] Step (9) Model training:

[0054] 9-1. Combine the contrastive learning loss function and the classification loss function as the overall loss function:

[0055]

[0056] where is the total loss finally used for backpropagation optimization, The hyperparameter is used to control the weight of the contrastive learning loss in the overall loss;

[0057] 9-2. Use the AdamW optimizer with a learning rate of 0.001 to update the parameters based on the overall loss function on the training set;

[0058] Step (10) Model prediction:

[0059] 10-1. After inputting the clause into the model, the classifier outputs a two-dimensional vector , where and respectively represent the original scores of the sample in the "risk-free" category and the "risky" category;

[0060] 10-2. The output of the classifier is normalized by the Softmax function to obtain a two-dimensional vector , where represents the predicted probability that the clause belongs to the "risk-free" category, represents the predicted probability that the clause belongs to the "risky" category, and the sum of the two probability values is 1, that is ;

[0061] 10-3. For each clause, if , then the clause is determined to be a risky clause, otherwise the clause is determined to be a risk-free clause.​

Claims

1. A method for identifying contract risk clauses based on natural language processing, characterized in that, It includes the following steps: Step 1: Extract a clause set from the contract dataset, where each clause is a sample data; Step 2: Uniformly preprocess the text in each sample data to generate sentence-level vectors; Step 3: Construct positive and negative sample pairs based on each sentence vector in the sentence-level vectors; Step 4: Use the contrastive learning method to learn the set of positive and negative sample pairs, and construct a classifier to output the clause recognition result.

2. The method for identifying contract risk terms based on natural language processing according to claim 1, wherein Step 1 specifically includes: extracting a clause set from the contract dataset , let be the number of clauses, and represent each clause as , that is, a sample data , where represents the clause number represents the text of the clause, and 0 represents no risk, 1 represents risk 3. The method for identifying contract risk terms based on natural language processing according to claim 2, wherein The specific content of Step 2 includes: Step 2.1: First, perform clause splitting on the clauses, separate each clause into several sentences according to punctuation marks, and remove meaningless characters, numbers, and HTML tags; then, based on the stop word list, remove the stop words that have no actual meaning, and then normalize the contract terms with ambiguity or variants. Finally, truncate or perform sliding window segmentation on the sentences to meet the maximum input length limit of the model; after preprocessing, each sample is represented as , where represents the sentence set, is the m-th sentence in , and m is the number of sentences contained in clause Step 2.

2. For each preprocessed clause , input it into the existing pre-trained large language model SBERT to obtain the vector representations of all sentences in the clause , ( ), where is the dimension of the sentence vector.

4. The method for identifying contract risk clauses based on natural language processing according to claim 3, characterized in that, The specific implementation process of constructing positive and negative sample pairs is as follows: Step 3.

1. For each sentence vector in calculate its attention weight; ​ Step 3.2: Weighted summation is performed on the sentence vectors according to the attention weights to obtain the vector representation of the clause of ; Step 3.3: Input the vector representation of each clause into the Transformer model twice to generate two different intermediate representations; regard the two intermediate vectors generated from the same clause as a positive sample pair, and regard the intermediate vectors generated from clauses of different risk categories as negative sample pairs.

5. The method for identifying contract risk terms based on natural language processing according to claim 4, wherein The specific implementation process of Step 4 is as follows: Step 4.1: Use the contrastive learning method to learn the set of positive and negative sample pairs, with the goal of maximizing the similarity between positive sample pairs and minimizing the similarity between negative sample pairs; among them, the similarity metric is calculated using cosine similarity, and the contrastive learning loss function uses the improved normalized temperature-scaled cross-entropy loss NT-Xent; Step 4.2, classifier construction: Input the vector representations of all clauses into a multi-layer perceptron for training. Use the cross-entropy loss function as the classification loss function to measure the difference between the predicted probability and the true label, which is used as the optimization objective for the classification task; Step 4.3: Combine the contrastive learning loss function and the classification loss function as the overall loss function, and use an optimizer for training to finally output the prediction scores corresponding to the two categories of risk-free and risky; after normalization, obtain the prediction probabilities of each category.

6. The method for identifying contract risk clauses based on natural language processing according to claim 5, wherein In Step 4.1, the improved normalized temperature-scaled cross-entropy loss NT-Xent is specifically calculated as follows: ； where is the contrastive learning loss function, is the temperature hyperparameter, is the similarity calculation function, and represent the positive sample and the negative sample of the sample respectively, is the set of negative samples of, is the number of positive sample pairs.

Images