A legal text analysis method based on a BERT model

Abstract

The application discloses a legal text analysis method based on a BERT model, and belongs to the technical field of data analysis, and comprises the following steps: S1, collecting original input samples, inputting the original input samples into a BERT model, and updating the BERT model; S2, generating an adversarial sample according to the original input samples; S3, inputting the original input samples and the adversarial sample into the updated BERT model, performing LMI adjustment, and obtaining a classification result of the original input samples. The application alleviates the performance decline caused by data imbalance, improves the prediction accuracy of low-frequency charges, effectively reduces the interference between easily-confused charges, and thus improves the overall prediction performance.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data analysis, and particularly relates to a legal text analysis method based on a BERT model. BACKGROUND

[0002] Legal charge prediction aims to predict the final charge according to the fact description in the case, and plays an important role in the legal intelligent system. However, due to the reality factors, the existing legal data sets mostly have the problem of sample imbalance, and the existing prediction methods mostly mainly focus on high-frequency charges, and pay less attention to low-frequency charges. For example, according to the first stage of the train single-label charge sample statistics in the CAIL2018 data set, the 10 common charges account for about 78.7%, and the 115 uncommon charges only account for 0.96%. In addition, due to the similarity between legal provisions, the complexity of the case, and the diversity of judicial interpretation and case facts, the confusion problem between charges is particularly prominent. The existing researches mostly mainly focus on the prediction of common charges, ignoring the processing problem of few-sample charges. SUMMARY

[0003] The application is proposed to solve the above problems, and provides a legal text analysis method based on a BERT model.

[0004] The technical scheme of the application is as follows:A legal text analysis method based on a BERT model comprises the following steps:

[0005] S1, collecting an original input sample, inputting the original input sample into a BERT model, and updating the BERT model;

[0006] S2, generating an adversarial sample according to the original input sample;

[0007] S3, inputting the original input sample and the adversarial sample into the updated BERT model, performing LMI adjustment, and obtaining the classification result of the original input sample.

[0008] Further, S1 comprises the following sub-steps:

[0009] S11, collecting an original input sample, and determining the prediction probability of the original input sample by the BERT model;

[0010] S12, generating a cross-entropy loss function according to the prediction probability of the original input sample;

[0011] S13, updating the parameters of the BERT model by using the gradient descent method based on the cross-entropy loss function.

[0012] Further, in S11, the prediction probability of the original input sample is calculated according to the following formula:

[0013] ;

[0014] In the formula, Represents the first of the original input samples The score of the class label, Represents the first of the original input samples The score of the class label, Represents an exponential function. Indicates the number of tags.

[0015] Furthermore, in S12, the cross-entropy loss function The expression is:

[0016] ;

[0017] In the formula, This represents the predicted probability of the original input sample. Represents the logarithmic function. Indicates the number of tags. Indicates the first Indicator functions for class tags.

[0018] Furthermore, in S13, the formula for calculating the updated BERT model parameters is as follows:

[0019] ;

[0020] In the formula, Represents the loss function. Indicates the first The parameters updated next time. Indicates the first The parameters updated next time. This represents the learning rate.

[0021] Furthermore, S2 includes the following sub-steps:

[0022] S21. Iteratively update the perturbation of the updated BERT model;

[0023] S22. Generate adversarial examples based on the iteratively updated perturbation results.

[0024] Furthermore, in S21, the calculation formula for iteratively updating the perturbation is as follows:

[0025] ;

[0026] In the formula, Indicates the first The disturbance of the next update Indicates the first The disturbance of the next update denotes a loss function, denotes a projection operation, denotes a step size, denotes a label corresponding to the original input sample, denotes a maximum amplitude of the perturbation, denotes a sign function, denotes a gradient of the loss function with respect to the perturbation of the th update, denotes an original input sample.

[0027] Further, in S22, the expression of the adversarial sample is:

[0028] ;

[0029] In the formula, denotes the perturbation of the th update, denotes an original input sample.

[0030] Further, in S3, the calculation formula of the classification result of the original input sample is:

[0031] ;

[0032] In the formula, denotes a score value of the original input sample.

[0033] The beneficial effects of the present application are: in view of the problems of sample imbalance and easy confusion between charges existing in the legal data set, the present application proposes a legal text analysis method based on a BERT model, which aims to enhance the robustness of the model, alleviate the performance decline caused by data imbalance, improve the prediction accuracy of low-frequency charges, effectively reduce the interference between easily confused charges, and thus improve the overall prediction performance. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is a flowchart of the legal text analysis method based on the BERT model;

[0035] Figure 2 is a schematic diagram of the function ;

[0036] Figure 3 is a case schematic diagram. DETAILED DESCRIPTION

[0037] The embodiments of the present application will be further described below in conjunction with the accompanying drawings.

[0038] As Figure 1As shown, this invention provides a legal text analysis method based on the BERT model, comprising the following steps:

[0039] S1. Collect the original input samples and input them into the BERT model to update the BERT model;

[0040] S2. Generate adversarial examples based on the original input samples;

[0041] S3. Input the original input samples and adversarial samples into the updated BERT model, perform LMI adjustment, and obtain the classification results of the original input samples.

[0042] In legal crime prediction tasks, it is often viewed as a text classification problem, with the goal of determining the crime based on the factual description of the case X={x1,x2,...,x...}. n}, where x i Given terms in vocabulary T, where n is the length of the word sequence, predict a unique crime label y∈Y, where Y ={y1,y2,...,y...} m} represents the set of all possible crime labels.

[0043] In this embodiment of the invention, S1 includes the following sub-steps:

[0044] S11. Collect the original input samples and determine the prediction probability of the BERT model for the original input samples;

[0045] S12. Generate the cross-entropy loss function based on the predicted probabilities of the original input samples;

[0046] S13. Based on the cross-entropy loss function, the gradient descent method is used to update the parameters of the BERT model.

[0047] In this embodiment of the invention, in S11, the predicted probability of the original input sample The calculation formula is:

[0048] ;

[0049] In the formula, Represents the first of the original input samples The score of the class label, Represents the first of the original input samples The score of the class label, Represents an exponential function. Indicates the number of tags.

[0050] In this embodiment of the invention, in S12, the cross-entropy loss function The expression is:

[0051] ;

[0052] wherein, denotes the prediction probability of the original input sample, denotes the logarithmic function, denotes the number of labels, denotes the indicator function of the th label.

[0053] In the embodiment of the application, in S13, the calculation formula of updating the parameters of the BERT model is:

[0054] ;

[0055] wherein, denotes the loss function, denotes the th updated parameter, denotes the th updated parameter, denotes the learning rate.

[0056] In the embodiment of the application, S2 comprises the following sub-steps:

[0057] S21, iteratively updating and perturbing the updated BERT model;

[0058] S22, generating an adversarial sample according to the result of the iterative updating and perturbing.

[0059] In the embodiment of the application, in S21, the calculation formula of the iterative updating and perturbing is:

[0060] ;

[0061] wherein, denotes the th updated perturbation, denotes the th updated perturbation, denotes the loss function, denotes the projection operation, denotes the step size, denotes the label corresponding to the original input sample, denotes the maximum amplitude of the perturbation, denotes the sign function, denotes the gradient of the loss function with respect to the th updated perturbation, denotes the original input sample.

[0062] In the embodiment of the application, in S22, the expression of the adversarial sample is:

[0063] ;

[0064] In the formula, represents the disturbance of the first updated, represents the original input sample.

[0065] In the embodiment of the present application, in S3, the calculation formula of the classification result of the original input sample is as follows:

[0066] ;

[0067] In the formula, represents the score value of the original input sample.

[0068] In the embodiment of the present application, in the legal charge classification task, there is a high similarity between some categories, which easily causes confusion and leads to classification errors. These confused categories often show a close original score (Logits) in the classification output layer. In order to reduce the influence of the confused categories on the classification accuracy, the present application proposes a method (LMI) based on adjusting the Logit value, which dynamically adjusts the current highest Logit value. Specifically, the highest Logits value L1 is adjusted by using the following formula: ; wherein, represents the adjusted highest Logit value, represents the current highest Logit value, represents the second highest Logit value, represents a stability factor, which is used to adjust the influence weight of the original highest Logit value and the dynamic adjustment part, so as to find a reasonable compromise between reducing similar category confusion and maintaining model stability.

[0069] As Figure 2 shown, the function dynamically adjusts the adjustment amplitude according to the difference value of .

[0070] The adjustment logic is as follows:

[0071] (1) The case of small difference value: when is small, it means that the two categories in the current classification result are highly confused. In this case, the value of is also small, avoiding misclassification caused by simply selecting the category represented by . At the same time, the value of is mainly determined by , so as to reduce the influence of confusion on classification decision.

[0072] (2) The case of moderate difference value: when When in the moderate range (e.g. 0.3 to 1.2), The value of the adjusted increases as the difference increases. At this time, the adjusted value will be pulled up, further widening the gap between and , and the discriminability of the two categories will be greater, thereby enhancing the discriminability of the classification model.

[0073] (3) Large difference: when is already large, indicating that the discriminability of the two categories is high and no additional adjustment is needed. At this time, The value of the adjusted gradually decreases as the difference increases, and the impact of the adjustment on tends to be stable.

[0074] Through this method, the adjusted Logits value can effectively reduce the classification errors caused by category confusion, improve the classification accuracy and robustness of the model. In summary, the LMI adjustment method proposed in the present application enhances the discriminability between confused categories by dynamically adjusting the maximum Logits value, and maintains stability when the category difference is large, effectively improving the performance of the classification task.

[0075] The performance of the model in the confused charge prediction task will be described in detail below with specific examples. As shown in Figure 3 , in this case, the defendant was ultimately convicted of the crime of "counterfeiting, altering, buying and selling of state organs' official documents, certificates, seals". However, due to the involvement of "certificates", "seals" and other noise data related to the crime in the case, traditional models are often prone to misjudgment as "counterfeiting company, enterprise, institution, people's organization seals crime". In contrast, the model proposed in the present application significantly reduces the incidence of such misjudgments through more effective feature extraction and contextual semantic understanding, and successfully predicts the correct charge. The specific results are shown in Table 1.

[0076] Table 1

[0077] Model Prediction results BERT CCP4CI Counterfeiting, altering, selling of state organs' documents, certificates, and seals Real label Counterfeiting, altering, selling of state organs' documents, certificates, and seals

[0078] To further verify the effectiveness of the model in dealing with the confusion of crime name labels, the present application selects several representative confusion crime name groups from the entire Charge_L dataset based on the misjudgment in the experiment for analysis. Among them, the medium and low frequency confusion crime name includes "destroying traffic tools" and "destroying traffic facilities", "destroying supervision order" and "abusing people under supervision", and "smuggling" and "smuggling of goods and articles prohibited by the state from import and export", the high frequency confusion crime name includes "producing and selling food not meeting safety standards", "producing and selling fake and inferior products" and "producing and selling toxic and harmful food", "abuse of power" and "negligence of duty", "opening gambling house" and "gambling", and "forging, altering, buying and selling of state organs' official documents, certificates and seals" and "forging company, enterprise, institution and people's group seals". As shown in Table 2, by analyzing the F1 value of these crime name groups, it can be found that whether in medium and low frequency or high frequency confusion crime name, the performance of the model is improved, which shows that the model has strong robustness and effectiveness in dealing with the confusion of crime name.

[0079] Table 2

[0080] Charge Type Mid-low frequency High frequency BERT CCP4CI 74.4092.73（⬆18.33%） 95.4996.51（⬆1.02%）

[0081] To further verify the advantages of the model in dealing with unbalanced data sets, the present application compares and analyzes the F1 values of different frequency categories of the entire Charge_L dataset. Specifically, according to the distribution of the number of samples of crime name, the crime name is divided into three categories: the number of samples less than or equal to 30 is low frequency crime name, the number of samples between 30 and 150 is medium frequency crime name, and the number of samples between 150 and 800 is high frequency crime name. The experimental results are shown in Table 3, the performance of the model in low frequency, medium frequency and high frequency crime name categories is better than that of the benchmark method, especially in the low frequency crime name category, the F1 value of the model is more than 43% higher than that of the benchmark method. This significant improvement shows that the model is effective in dealing with the problem of unbalanced sample distribution.

[0082] Table 3

[0083] Charge Type Low frequency Medium frequency High frequency Charge Number 20 57 116 BERT CCP4CI 40.3983.75（⬆43.36%） 92.7895.07（⬆2.29%） 96.8496.93（⬆0.09%）

[0084] Those skilled in the art will realize that the embodiments described herein are for the purpose of helping the reader understand the principles of the present application and should be understood as not limiting the scope of protection of the present application to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations according to the technical inspiration disclosed in the present application without departing from the essence of the present application, and these modifications and combinations are still within the scope of protection of the present application.

Claims

1. A method for analyzing legal text based on a BERT model, characterized in that, The method comprises the following steps: S1, collecting an original input sample, inputting the original input sample into a BERT model, and updating the BERT model; S2, generating an adversarial sample according to the original input sample; S3, inputting the original input sample and the adversarial sample into the updated BERT model, performing LMI adjustment, and obtaining a classification result of the original input sample; The S1 comprises the following sub-steps: S11, collecting an original input sample, and determining a prediction probability of the original input sample by the BERT model; S12, generating a cross-entropy loss function according to the prediction probability of the original input sample; S13, updating the parameters of the BERT model by using a gradient descent method based on the cross-entropy loss function; In S11, the predicted probability of the original input sample The calculation formula is: ； wherein, denotes the i-th original input sample, score value of the class label, denotes the i-th original input sample, score value of the class label, denotes an exponential function, denotes the number of labels; In the S12, the cross-entropy loss function The expression is: ； wherein, denotes a predicted probability of an original input sample, denotes a logarithm function, denotes a number of labels, denotes an indicator function for the class label. In the S13, the calculation formula for updating the parameters of the BERT model is: ； wherein denotes a loss function, denotes the updated parameter, denotes the updated parameter, denotes a learning rate; In the S3, the calculation formula of the classification result of the original input sample is: ​ ； In the formula, denotes the score value of the original input sample; The current highest Logit value is dynamically adjusted, and the highest Logit value L1 is adjusted by using the following formula: ; wherein, represents the adjusted highest Logit value, represents the current highest Logit value, represents the current second highest Logit value, represents a stability factor; Function The function is to dynamically adjust the adjustment range according to the difference value of , and the adjustment logic is as follows: (1) the difference is less than the first set threshold: when the difference is less than the first set threshold, it means that the two classes in the current classification result are highly confused, in which case, the value of the difference is less than the corresponding threshold, so as to avoid misclassification caused by simply selecting the class represented by the difference, the value of the difference is mainly determined by , so as to reduce the influence of confusion on the classification decision. (2) the case where the difference is greater than or equal to the first set threshold and less than or equal to the second set threshold: when greater than or equal to the first set threshold and less than or equal to the second set threshold, the value of the difference increases, at this time, the adjusted value will be pulled up, thereby further expanding the gap between and , and the discriminability of the two categories will be greater, thereby enhancing the discriminability of the classification model; (3) The case where the difference is greater than the second set threshold value: when When the difference is greater than the second set threshold value, it indicates that the degree of distinction between the two categories is greater than the degree of distinction threshold value, and no additional adjustment is needed, at which time, The value of the difference gradually decreases as the difference increases.

2. The BERT model-based legal text analysis method according to claim 1, characterized in that, The S2 comprises the following sub-steps: S21, performing iterative update disturbance on the updated BERT model; S22, generating an adversarial sample according to the iterative update disturbance result. 3.The BERT model-based legal text analysis method of claim 2, wherein, In the S21, the calculation formula for performing the iterative update disturbance is: ； wherein denotes the perturbation of the th update, denotes the perturbation of the th update, denotes the loss function, denotes the projection operation, denotes the step size, denotes the label corresponding to the original input sample, denotes the maximum magnitude of the perturbation, denotes the sign function, denotes the gradient of the loss function with respect to the perturbation of the th update, denotes the original input sample. 4.The BERT model-based legal text analysis method according to claim 2, characterized in that, In the S22, the expression of the anti-sample is: ； wherein denotes the perturbation of the denotes the original input sample.

Images