Law decision prediction method based on BERT model and keyword extraction

Abstract

The invention relates to the field of natural language processing technology and law decision prediction (LJP), and provides a method based on a BERT model, keyword extraction and a cross attention mechanism. Specifically, the method comprises the steps that firstly, case fact description is input into a keyword extraction model based on Transform, legal fact keywords are extracted, and the legal fact keywords are input into a BERT model to obtain vector representation; thirdly, fusing the forensic semantics into the model by utilizing a cross attention mechanism, so as to predict criminal name prediction, law article prediction (classification task) and criminal period prediction (regression task); according to the method, the keywords related to the context are extracted and embedded into the BERT model, and meanwhile, a cross attention mechanism is adopted to be brought into the forensic semantics, so that the problem that key event information is difficult to accurately capture in a traditional method is effectively solved, and the prediction accuracy of each subtask of the LJP is improved.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing technology and legal judgment prediction, and specifically relates to a method based on a BERT model, keyword extraction and legal article constraint. Background Art

[0002] Legal Judge Prediction (LJP) is an important downstream subtask in legal judgment decision-making. It aims to predict the judgment results based on the factual description in the text, including relevant legal provisions, charges and sentences. Usually, legal cases often involve a large number of legal provisions, precedents, facts and evidence, which makes the judgment process very complicated and full of uncertainty. In addition, judges may be influenced by their personal concepts, experience and judgment when making judgments, resulting in obvious differences in the judgment results of similar cases.

[0003] At the same time, using data science and machine learning techniques to analyze relevant information about legal cases and predict the corresponding court decisions can improve the efficiency of the judicial system and reduce the burden of court work, especially when dealing with a large number of cases. Predictive models can help reduce the interference of subjective factors, promote more consistent decisions in similar cases, and improve judicial fairness. At the same time, legal decisions help lawyers, parties, and judges better understand the direction of the case, providing strong support for the formulation of defense and judgment strategies.

[0004] In the legal field, LJP tasks can improve the consistency of judgments, help achieve social justice, ensure similar cases get similar legal results, and the use of prediction models can promote the stability and predictability of the rule of law and enhance the public's trust in the judicial system. Since court resources are limited, prediction can be used to allocate resources more effectively and focus on cases that have a greater impact on the judgment results. In general, the research on legal judgment prediction aims to improve the efficiency, consistency and fairness of the judicial system with the help of modern technical means, while providing better decision-making support for legal practitioners.

[0005] Identifying facts is the most basic step in making judgments, so detecting events in legal documents is important for legal case analysis tasks. However, existing legal event detection (LED) datasets only cover incomplete event types and have limited annotation data, which restricts the development of LED methods and their downstream applications. The LEVEN Chinese dataset proposed by Yao et al. earlier contains 100,000 legal event mentions, which is dozens of times larger than the previous LED dataset and is currently the largest Chinese legal event detection dataset. However, there are still inevitably some event types with limited instances, and there is a large gap between the performance on low-frequency event types and the overall average performance. At the same time, many prediction errors in the literature are caused by incorrectly capturing contextual information. The problems of these studies can be summarized as follows: (1) Failure to extract fine-grained key events of the case; (2) Failure to predict the judgment based on the extracted event information (rather than the entire statement of facts). Summary of the invention

[0006] In order to solve the problems existing in the above prior art, the present invention proposes a BERT model and keyword extraction method, which includes: obtaining a CAIL-2018 case description dataset and a code and provision dataset, preprocessing the data, inputting the preprocessed case description data into a trained keyword extraction model, obtaining the keywords of the case description, inputting the keywords and legal provision data into the BERT model respectively to obtain corresponding word vectors and matching them using a cross-attention mechanism, and finally performing sentence prediction (regression task), legal provision prediction (classification task), and crime prediction (classification task) according to the matching results; the keyword extraction model is a model trained for keyword search capabilities using a Transformer-based and additional training paradigm after preprocessing the LEVEN dataset;

[0007] The pre-training of the keyword extraction model and the subsequent training of the LJP subtask include the following processes:

[0008] S1: Input the case description in the original LEVEN dataset into the Transformer encoder to generate the corresponding semantic vector;

[0009] S2: In the decoding process, a set of learnable control codes is introduced. Specifically, each step of the decoder input contains the vector representation of the word generated at the previous moment, the position embedding and the corresponding control code embedding. This method guides the generation of key phrases of specific categories through control codes, improving the model's ability to generate diversity;

[0010] S3: When the model generates candidate key phrases, it uses the K-step target allocation method to generate K candidate words for each control code and records the predicted probability distribution of each word. Then, it uses the bipartite graph matching algorithm to match the candidate keywords predicted by the model with the real key phrases to ensure that each real keyword has a unique matching candidate word;

[0011] S4: Define the loss function training model according to the correspondence between each control code and the target;

[0012] S5: Get the CAIL-2018 dataset and input each fact description into the trained keyword extraction model to obtain the corresponding keywords;

[0013] S6: Input the keyword into the BERT model to obtain the vector representation of the keyword, and pass the vector into the cross-attention layer. The result represents the contextual representation of the keyword in the fact description.

[0014] S7: Input the legal code into the BERT model to obtain the corresponding vector representation, and pass the vector into the cross-attention layer. The result represents the contextual representation of the legal code;

[0015] S8: Perform matching calculations on the results obtained in steps S6 and S7 to obtain the attention weights of the code provisions on the keywords describing the facts, thereby incorporating the code provisions into the model. Finally, the context description of the code provisions with the highest matching level is input into the softmax layer together with the context representation of the fact description to perform the two classification tasks of crime prediction and law prediction, and the cross-entropy loss function is used to train the model. For the regression task of sentence prediction, the above two context representations are input into the linear layer to predict the corresponding results, and the logarithmic distance function is used as the loss function.

[0016] Preferably, the process of preprocessing the data includes: taking the original LEVEN data set, processing the original data set, inputting the case description into the Transformer encoder, generating a vector representation corresponding to each word, and representing the fact statement Q as a sequence constructed by sentence words:

[0017] Q={a1,…,a n}

[0018] Where Q represents the input factual statement, a i Represents the vector representation of the i-th word in the fact statement.

[0019] Preferably, the process of inputting the vector representation Q of the fact description into the Transformer decoder and using a set of learnable control codes as additional decoder input comprises:

[0020] Step 1: Set the number of control codes N to a large enough value to ensure that all keywords in each input fact statement can be predicted. At the same time, introduce a special marker φ to represent "no keyword" to fill in those control codes that do not need to generate keywords. Among the N candidate keywords generated by the model, delete the items marked as φ to obtain the actual set of keywords related to the input fact statement. In this way, the appropriate number of keywords can be automatically determined based on the model output without knowing the exact number of input keywords in advance.

[0021] Step 2: For the i-th control code, input it into the decoder at time step t, and its representation is:

[0022]

[0023] in is the vector representation of the predicted word at the previous moment for the i-th control code at time step t, is the word predicted by the ith control code at time step t, is the sinusoidal position embedding at time step t, c i is the embedding representation of the i-th control code, which is used to guide the model to generate different key phrases.

[0024] 1. Preferably, for each of the K candidate keywords generated by the control code, the predicted probability distribution thereof during the decoding process is recorded, and at the same time, a binary match is used between the real keyword and the predicted keyword to ensure that each target element has a unique matching item, and to ensure that in the candidate keyword sequence generated by each control code, the item that best matches the real keyword is selected, and the process includes:

[0025] Step 1: Generate K words for each control code, and then collect the corresponding predicted probability distribution of each step, expressed as:

[0026] P={P n} n＝1,…,N

[0027]

[0028] in represents the predicted distribution of control code n at time step t, N represents the number of control codes, and K represents the number of candidate keywords generated for each control code;

[0029] Step 2: Find a bipartite match between the real keyword and the predicted keyword (the result of the bipartite match is an alignment relationship, ensuring that each target keyword has a unique match and no repeated keywords appear in the prediction). Assuming that the number of predefined control codes N is greater than the number of target keywords, in order to avoid generating repeated keywords, the target keyword set is completed to N elements, where the empty mark (φ) indicates that there is no corresponding keyword. In order to find a bipartite match between the real keyword set and the predicted set, define the permutation with the lowest search cost:

[0030]

[0031] Among them, π is a permutation, which belongs to the ∏(N) space, ∏(N) is the set of all permutations of length N, y n represents the nth target keyword sequence, P π(n) represents the distribution of the nth predicted keyword sequence after arrangement, C match (y n ,P π(n) ) is the pairwise matching cost between the target keyword sequence and the predicted sequence distribution, where the matching cost takes into account the category prediction and is defined as follows:

[0032]

[0033] Where S = min(K,|y n |) represents the minimum shared length between the target sequence and the predicted sequence. The purpose is to limit the number of keywords considered when calculating the matching cost to ensure the effectiveness and rationality of the calculation. n | indicates the length of the target keyword sequence, Expressive words exist The probability in .

[0034] Preferably, according to the corresponding relationship between each control code and the target, the formula for defining the loss function training model is:

[0035]

[0036] Among them, loss(θ) represents the corresponding loss function value, N is the number of samples, |y n | is the length of the nth target keyword sequence, It is the predicted probability distribution using teacher forcing, which is used to ensure that the model matches the target keywords when generating training.

[0037] Preferably, the process of obtaining the CAIL 2018 dataset and inputting each fact description into the trained keyword extraction model to obtain the corresponding keyword includes:

[0038] Step 1: Take the CAIL 2018 dataset. Here, the maximum length allowed for the model is set to 512 characters. Fact descriptions with a length of less than 512 characters are padded, and fact descriptions with a length of more than 512 characters are truncated from the end of the sentence.

[0039] Step 2: From step 1, a data set that meets the conditional length will be obtained. This data set is input into the keyword extraction model to obtain the corresponding keyword set.

[0040] Preferably, the process of inputting the keyword into the BERT model to obtain a vector representation of the keyword, and passing the vector into the cross-attention layer, and obtaining a result representing the contextual representation of the keyword in the fact description includes:

[0041] Step 1: For the keyword set D = {x1, x2, ..., x n}, input each word into the pre-trained BERT encoder for encoding and obtain the corresponding vector representation:

[0042] {h1,h2,…,h n}=BERT(x1,x2,…,x n )

[0043] H={h1,h2,…,h n} is the vector sequence of the keyword set, n is the number of keywords;

[0044] Step 2: Input H into the fully connected layer in the cross attention layer to obtain U = {u1,u,…,u n}:

[0045] u i =tanh(W i h i +b i ),i∈[1,n]

[0046] Step 3: i Use the softmax function to obtain the normalized importance weight α i , which measures the importance of a word:

[0047]

[0048] Step 4: Add α i With h i Perform point multiplication and add to get the corresponding hidden state weighted sum, r represents the output of the cross attention layer, that is, the context representation of the corresponding fact description:

[0049]

[0050] Preferably, the code is input into the BERT model to obtain a corresponding vector representation, and the vector is passed into the cross-attention layer, and the result obtained represents the contextual representation of the keyword in the fact description. The process includes:

[0051] Step 1: Use the same encoder to encode the character sequence of the code to obtain the corresponding vector H′={h′1,h′2,…,h′ len}, where len represents the length of the legal article;

[0052] Step 2: Input H′ into the fully connected layer to obtain U′={u′1,u′2,…,u′ len}:

[0053] u′ j =tanh(W j 'h' j +b′ j ),j∈[1,len]

[0054] Step 3: For u′ j Use the softmax function to obtain the normalized importance weight α′ j , which measures the importance of a word:

[0055]

[0056] Step 4: Set α′ j and h′ j Perform point multiplication and add to get the corresponding hidden state weighted sum, c represents the output of the cross attention layer, that is, the context representation of the corresponding code:

[0057]

[0058] Through the above four steps, the context representation of each code will be obtained: C = {c1, c2, ..., c l}, l represents the number of articles in the code.

[0059] Preferably, the results obtained in steps S6 and S7 are matched and calculated to obtain the attention weights of the code provisions on the keywords describing the facts, thereby incorporating the code provisions into the model. Finally, the context description of the code provisions with the highest matching level is input into the softmax layer together with the context representation of the fact description to perform the two classification tasks of crime prediction and law prediction, and the model is trained using the cross entropy loss function. For the regression task of sentence prediction, the above two context representations are input into the linear layer to predict the corresponding results. The process of using the logarithmic distance function as the loss function includes:

[0060] Step 1: Incorporate the legal code into the model and obtain the attention weight of the legal code to the keywords describing the facts:

[0061] W ij =f(r i c j ),i∈[1,n],j∈[1,l]

[0062]

[0063] Among them, f() represents the inner product function, β ij Indicates the attention weight of the keywords in the code of law on the description of facts;

[0064] Step 2: Use a scaled dot product attention mechanism to weigh the relative importance of different code provisions:

[0065]

[0066] Among them, ρ represents the attention weight of the code provisions, and the weight matrix

[0067] Step 3: Combine β and ρ through the inner product f() to get the final attention weight γ, and then add γ to r i Perform inner product and perform softmax operation on the result to get q i , and finally all q i Concatenate them into a vector S and use the maximum pooling layer to get the final representation O:

[0068] γ=f(β,ρ)

[0069] q i =softmax(f(γ,r i ))

[0070] S=[q1,q2,…,q l ]

[0071] O = maxpooling(S)

[0072] Step 4: Input the final representation O into the softmax layer to get the final result:

[0073]

[0074] Here t and b tThey are respectively a trainable matrix and a bias. However, since crime prediction and law prediction are two classification tasks, two softmax operations are performed separately. For sentence prediction, the final representation O of the maximum pooling layer is input into a linear layer and calculated using the ReLU activation function. The formula is as follows:

[0075]

[0076] Among them, ω1, ω0, b0, and b1 are all trainable vectors and biases.

[0077] Each classification task here uses the cross entropy function as the loss function, and the regression task uses the logarithmic distance function as the loss function:

[0078] For the law prediction task, the cross entropy function is used as the loss function, and the formula is as follows:

[0079]

[0080] Among them, |Law| is the number of legal categories, It is the true label of the law. is the model prediction probability distribution;

[0081] For the crime prediction task, the cross entropy function is used as the loss function, and the formula is as follows:

[0082]

[0083] Among them, |Crime| is the number of crime categories, It is the true label of the crime. is the model prediction probability distribution;

[0084] For the sentence prediction task, the logarithmic distance function is used as the loss function, and the formula is as follows:

[0085]

[0086] in, is the sentence predicted by the model, y term is the actual sentence, ∈ is a small value used to avoid errors in logarithmic operations, usually 1e-6.

[0087] Finally, the loss function is defined as:

[0088] Loss(θ)=λ ta Loss(t a )+λ tc Loss(t c )+λ tp Loss(t p )

[0089] Here t a ,t c ,t p Represent three subtasks, λ ta , tc , tp It represents the weight coefficient of each subtask loss, and Loss(θ) represents the total loss function value.

[0090] The present invention uses the BERT model to obtain context-related word embeddings, solving the problem that previous methods cannot fully utilize context semantics; the present invention pre-trains a keyword extraction model using the LEVEN dataset, so that it can more accurately find the corresponding trigger words on the CAIL 2018 dataset; the present invention uses the maximum pooling technology on the BERT model to perform further feature extraction on the extracted keywords, thereby improving the accuracy of the final result; the present invention incorporates the semantics of legal articles into the model, and obtains the correlation score by matching the features of the fact description and the features of the legal articles, thereby improving the accuracy of the final legal prediction task. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] Figure 1 It is the overall flow chart of the present invention;

[0092] Figure 2 It is a structural diagram of the keyword extraction model of the present invention;

[0093] Figure 3 This is a structural diagram of the legal decision prediction model of the present invention. DETAILED DESCRIPTION

[0094] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0095] A BERT model and keyword extraction method, such as Figure 1The method comprises: obtaining a case description dataset and a code and provision dataset in CAIL 2018, preprocessing the data, inputting the preprocessed case description data into a trained keyword extraction model to obtain keywords of the case description, respectively inputting the keywords and legal provision data into a BERT model to obtain corresponding word vectors and matching them using a cross-attention mechanism, and finally performing sentence prediction (regression task), legal provision prediction (classification task), and crime prediction (classification task) according to the matching results; the keyword extraction model is a model that is trained for keyword search capabilities using a Transformer-based and new training paradigm after preprocessing the LEVEN dataset.

[0096] S1: Process the original LEVEN dataset, input the case description into the Transformer encoder, and generate the corresponding semantic vector representation;

[0097] S2: In the decoding process, a set of learnable control codes is introduced. Specifically, each step of the decoder input contains the vector representation of the word generated at the previous moment, the position embedding and the corresponding control code embedding. This method guides the generation of key phrases of specific categories through control codes, improving the model's ability to generate diversity;

[0098] S3: When the model generates candidate key phrases, it uses the K-step target allocation method to generate K candidate words for each control code and records the predicted probability distribution of each word. Then, it uses the bipartite graph matching algorithm to match the candidate keywords predicted by the model with the real key phrases to ensure that each real keyword has a unique matching candidate word;

[0099] S4: Define the loss function training model according to the correspondence between each control code and the target;

[0100] S5: Get the CAIL-2018 dataset and input each fact description into the trained keyword extraction model to obtain the corresponding keywords;

[0101] S6: Input the keyword into the BERT model to obtain the vector representation of the keyword, and pass the vector into the cross-attention layer. The result represents the contextual representation of the keyword in the fact description.

[0102] S7: Input the legal code into the BERT model to obtain the corresponding vector representation, and pass the vector into the cross-attention layer. The result represents the contextual representation of the legal code;

[0103] S8: Perform matching calculations on the results obtained in steps S6 and S7 to obtain the attention weights of the code provisions on the keywords describing the facts, thereby incorporating the code provisions into the model. Finally, the context description of the code provisions with the highest matching level is input into the softmax layer together with the context representation of the fact description to perform the two classification tasks of crime prediction and law prediction, and the cross entropy loss function is used to train the model. For the regression task of sentence prediction, the above two context representations are input into the linear layer to predict the corresponding results, and the logarithmic distance function is used as the loss function.

[0104] The LEVEN dataset and CAIL 2018 are used as the original legal event datasets. The datasets are divided into three parts. The training set of the LEVEN dataset contains 5301 legal events, the test set contains 1585 legal events, and the validation set contains 1230 legal events. The training set of the CAIL 2018 dataset contains 154592 legal judgments, the test set contains 32508 legal judgments, and the validation set contains 17131 legal judgments.

[0105] like Figure 2 As shown in the figure, this is a keyword extraction model. It takes the original LEVEN dataset, processes the original dataset, inputs the case description into the Transformer encoder, generates a vector representation corresponding to each word, and represents the fact statement Q as a sequence constructed by sentence words:

[0106] Q={a1,…,a n}

[0107] Where Q represents the input factual statement, a i Represents the vector representation of the i-th word in the fact statement.

[0108] The vector representation Q of the fact description is input into the Transformer decoder, and a set of learnable control codes are used as additional decoder inputs: the number of control codes N is set to a large enough value to ensure that all keywords in each input fact statement can be predicted. At the same time, a special marker φ is introduced to indicate "no keyword" to fill in those control codes that do not need to generate keywords. Among the N candidate keywords generated by the model, the items marked as φ are deleted to obtain the actual set of keywords related to the input fact statement. In this way, the appropriate number of keywords can be automatically determined based on the model output without knowing the exact number of input keywords in advance. At the same time, for the i-th control code, it is input into the decoder at time step t, and its representation is:

[0109]

[0110] in is the vector representation of the predicted word at the previous moment for the i-th control code at time step t, is the word predicted by the ith control code at time step t, is the sinusoidal position embedding at time step t, c i is the embedding representation of the i-th control code, which is used to guide the model to generate different key phrases.

[0111] After that, for each control code to generate K candidate keywords, record its predicted probability distribution during the decoding process, and use binary matching between the real keyword and the predicted keyword to ensure that each target element has a unique match, so that in the candidate keyword sequence generated by each control code, the one that best matches the real keyword is selected. Each control code generates K words, and then collects the corresponding predicted probability distribution of each step, which is expressed as:

[0112] P={P n} n＝1,…,N

[0113]

[0114] in represents the predicted distribution of control code n at time step t, N represents the number of control codes, and K represents the number of candidate keywords generated by each control code; then, a bipartite match is found between the real keyword and the predicted keyword (the result of the bipartite match is an alignment relationship, which ensures that each target keyword has a unique match and no repeated keywords appear in the prediction). Assuming that the predefined number of control codes N is greater than the number of target keywords, in order to avoid generating repeated keywords, the target keyword set is completed to N elements, where the empty mark (φ) indicates that there is no corresponding keyword. In order to find a bipartite match between the real keyword set and the predicted set, the permutation with the lowest search cost is defined:

[0115]

[0116] Among them, π is a permutation, which belongs to the ∏(N) space, ∏(N) is the set of all permutations of length N, y n represents the nth target keyword sequence, P π(n) represents the distribution of the nth predicted keyword sequence after arrangement, C match (y n ,P π(n) ) is the pairwise matching cost between the target keyword sequence and the predicted sequence distribution, where the matching cost takes into account the category prediction and is defined as follows:

[0117]

[0118] Where S = min(K,|y n |) represents the minimum shared length between the target sequence and the predicted sequence. The purpose is to limit the number of keywords considered when calculating the matching cost to ensure the effectiveness and rationality of the calculation. n | indicates the length of the target keyword sequence, Expressive words exist The probability in .

[0119] Finally, according to the correspondence between each control code and the target, the formula for defining the loss function training model is:

[0120]

[0121] Among them, loss(θ) represents the corresponding loss function value, N is the number of samples, |y n | is the length of the nth target keyword sequence, It is the predicted probability distribution using teacher forcing, which is used to ensure that the model matches the target keywords when generating training.

[0122] like Figure 3 As shown in Figure 1, the CAIL 2018 dataset is taken out. Here, the maximum length allowed for processing by the model is set to 512. Fact descriptions with a length less than 512 are padded, and those with a length greater than 512 are truncated from the end of the sentence. The preprocessed dataset is input into the previously trained keyword extraction model (Key Word BERT) to generate the corresponding keyword set D = {x1, x2, …, x n}, and then input it into the new BERT model to get the corresponding vector H:

[0123] H={h1,h2,…,h n}=BERT(x1,x2,…,x n )

[0124] Input H into the fully connected layer in the cross attention layer to obtain U = {u1,u,…,u n}:

[0125] u i =tanh(W i h i +b i ),i∈[1,n]

[0126] After that, i Use the softmax function to obtain the normalized importance weight α i , which measures the importance of a word:

[0127]

[0128] α i With h i Perform point multiplication and add to get the corresponding hidden state weighted sum, r represents the output of the cross attention layer, that is, the context representation of the corresponding fact description:

[0129]

[0130] Similarly, the code of law is input into the BERT model to obtain the corresponding vector representation H′={h′1,h′2,…,h′ len}, where len represents the length of the legal article;

[0131] Input H′ into the fully connected layer to obtain U′={u′1,u′2,…,u′ len}:

[0132] u′ j =tanh(W j 'h' j +b′ j ),j∈[1,len]

[0133] Then u′ j Use the softmax function to obtain the normalized importance weight α′ j , which measures the importance of a word:

[0134]

[0135] α′ j and h′ j Perform point multiplication and add to get the corresponding hidden state weighted sum, c represents the output of the cross attention layer, that is, the context representation of the corresponding code:

[0136]

[0137] Following the above steps, we can get the context representation of each code: C = {c1, c2, ..., c l}, l represents the number of articles in the code.

[0138] After that, the code of laws is incorporated into the model, and the attention weight of the code of laws on the keywords describing the facts is obtained:

[0139] W ij =f(r i c j ),i∈[1,n],j∈[1,l]

[0140]

[0141] Among them, f() represents the inner product function, β ij It represents the attention weight of the code of law to the keywords describing the facts, and adopts the scaled dot product attention mechanism to measure the relative importance of different code of law:

[0142]

[0143] Among them, ρ represents the attention weight of the code provisions, and the weight matrix Then, β and ρ are combined through the inner product f() to obtain the final attention weight γ, and then γ is combined with r i Perform inner product and perform softmax operation on the result to get q i , and finally all q i Concatenate them into a vector S and use the maximum pooling layer to get the final representation O:

[0144] γ=f(β,ρ)

[0145] q i =softmax(f(γ,r i ))

[0146] S=[q1,q2,…,q l ]

[0147] O = maxpooling(S)

[0148] Finally, the final representation O is input into the softmax layer to get the final result:

[0149]

[0150] Here t and b t They are respectively a trainable matrix and a bias. However, since crime prediction and law prediction are two classification tasks, two softmax operations are performed separately. For sentence prediction, the final representation O of the maximum pooling layer is input into a linear layer and calculated using the ReLU activation function. The formula is as follows:

[0151]

[0152] Among them, ω1, ω0, b0, and b1 are all trainable vectors and biases.

[0153] Each classification task here uses the cross entropy function as the loss function, and the regression task uses the logarithmic distance function as the loss function:

[0154] For the law prediction task, the cross entropy function is used as the loss function, and the formula is as follows:

[0155]

[0156] Among them, |Law| is the number of legal categories, It is the true label of the law. is the model prediction probability distribution;

[0157] For the crime prediction task, the cross entropy function is used as the loss function, and the formula is as follows:

[0158]

[0159] Among them, |Crime| is the number of crime categories, It is the true label of the crime. is the model prediction probability distribution;

[0160] For the sentence prediction task, the logarithmic distance function is used as the loss function, and the formula is as follows:

[0161]

[0162] in, is the sentence predicted by the model, y term is the actual sentence, ∈ is a small value used to avoid errors in logarithmic operations, usually 1e-6.

[0163] Finally, the loss function is defined as:

[0164] Loss(θ)=λ ta Loss(t a )+λ tc Loss(t c )+λ tp Loss(t p )

[0165] Here t a ,t c ,t p Represent three subtasks, λ ta , tc , tp It represents the weight coefficient of each subtask loss, and Loss(θ) represents the total loss function value.

[0166] Since there are three subtasks, the P, R, and F1 indicators are used to evaluate the model for the law prediction task and the crime prediction task. For the sentence prediction, the logarithmic distance function used is used as the loss value as the evaluation indicator.

[0167] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A BERT model and keyword extraction method, characterized in that: include: First, obtain the CAIL-2018 case description dataset and the code and regulations dataset and preprocess them. Input the preprocessed case description data into the trained keyword extraction model to extract the keywords of the case description; then, input the keywords and legal provisions data into the BERT model respectively to obtain the corresponding vector representations, and match them through the cross-attention mechanism. Finally, according to the matching results, perform sentence prediction (regression task), legal provision prediction (classification task), and crime prediction (classification task). Among them, the keyword extraction model is based on the preprocessed Large-Scale Chinese Legal Event Detection Dataset (LEVEN) dataset, and its keyword recognition ability is trained through the Transformer model and additional training paradigms; The pre-training of the keyword extraction model and the training of the LJP subtask include the following processes: S1: Process the original LEVEN dataset, input the case description into the Transformer encoder, and generate the corresponding semantic vector representation; S2: In the decoding process, a set of learnable control codes is introduced. Specifically, each step of the decoder input contains the vector representation of the word generated at the previous moment, the position embedding and the corresponding control code embedding. This method guides the generation of key phrases of specific categories through control codes, improving the model's ability to generate diversity; S3: When the model generates candidate key phrases, it uses the K-step target allocation method to generate K candidate words for each control code and records the predicted probability distribution of each word. Then, it uses the bipartite graph matching algorithm to match the candidate keywords predicted by the model with the real key phrases to ensure that each real keyword has a unique matching candidate word; S4: Define the loss function training model according to the correspondence between each control code and the target; S5: Get the CAIL-2018 dataset and input each fact description into the trained keyword extraction model to obtain the corresponding keywords; S6: Input the keyword into the BERT model to obtain the vector representation of the keyword, and pass the vector into the cross-attention layer. The result represents the contextual representation of the keyword in the fact description. S7: Input the legal code into the BERT model to obtain the corresponding vector representation, and pass the vector into the cross-attention layer. The result represents the contextual representation of the legal code; S8: Perform matching calculations on the results obtained in steps S6 and S7 to obtain the attention weights of the code provisions on the keywords describing the facts, thereby incorporating the code provisions into the model. Finally, the context description of the code provisions with the highest matching level is input into the softmax layer together with the context representation of the fact description to perform the two classification tasks of crime prediction and law prediction, and the cross-entropy loss function is used to train the model. For the regression task of sentence prediction, the above two context representations are input into the linear layer to predict the corresponding results, and the logarithmic distance function is used as the loss function.

2. A BERT model and keyword extraction method according to claim 1, characterized in that: The process of preprocessing the data includes: taking the original LEVEN dataset, processing the original dataset, inputting the case description into the Transformer encoder, generating a vector representation corresponding to each word, and representing the fact statement Q as a sequence constructed by sentence words: Q={a1,...,a n } Where Q represents the input factual statement, a i Represents the vector representation of the i-th word in the fact statement.

3. A BERT model and keyword extraction method according to claim 2, characterized in that: The process of feeding the fact description vector representation Q into the Transformer decoder, while using a set of learnable control codes as additional decoder inputs, includes: Step 1: Set the number of control codes N to a sufficiently large value to ensure that all keywords in each input fact statement can be predicted. At the same time, introduce a special marker φ to represent "no keyword" to fill in those control codes that do not need to generate keywords. Among the N candidate keywords generated by the model, delete the items marked as φ to obtain the actual set of keywords related to the input fact statement. In this way, the appropriate number of keywords can be automatically determined based on the model output without knowing the exact number of input keywords in advance. Step 2: For the i-th control code, input it into the decoder at time step t, and its representation is: in is the vector representation of the predicted word at the previous moment for the i-th control code at time step t, is the word predicted by the ith control code at time step t, is the sinusoidal position embedding at time step t, c i is the embedding representation of the i-th control code, which is used to guide the model to generate different key phrases.

4. A BERT model and keyword extraction method according to claim 3, characterized in that: For each of the K candidate keywords generated by the control code, record its predicted probability distribution during the decoding process, and use binary matching between the real keyword and the predicted keyword to ensure that each target element has a unique match, and to ensure that the candidate keyword sequence generated by each control code selects the one that best matches the real keyword. The process includes: Step 1: Generate K words for each control code, and then collect the corresponding predicted probability distribution of each step, expressed as: P={P n } n＝1，...，N in represents the predicted distribution of control code n at time step t, N represents the number of control codes, and K represents the number of candidate keywords generated for each control code; Step 2: Find a bipartite match between the real keyword and the predicted keyword (the result of the bipartite match is an alignment relationship, ensuring that each target keyword has a unique match and no repeated keywords appear in the prediction). Assuming that the number of predefined control codes N is greater than the number of target keywords, in order to avoid generating repeated keywords, the target keyword set is completed to N elements, where the empty mark (φ) indicates that there is no corresponding keyword. In order to find a bipartite match between the real keyword set and the predicted set, define the permutation with the lowest search cost: Among them, π is a permutation, which belongs to the Π(N) space, Π(N) is the set of all permutations of length N, y n represents the nth target keyword sequence, P π(n) represents the distribution of the nth predicted keyword sequence after arrangement, C match (y n , P π(n) ) is the pairwise matching cost between the target keyword sequence and the predicted sequence distribution, where the matching cost takes into account the category prediction and is defined as follows: Where S = min(K, |y n |) represents the minimum shared length between the target sequence and the predicted sequence. The purpose is to limit the number of keywords considered when calculating the matching cost to ensure the effectiveness and rationality of the calculation. n | indicates the length of the target keyword sequence, Expressive words exist The probability in .

5. A BERT model and keyword extraction method according to claim 4, characterized in that: According to the correspondence between each control code and the target, the formula for defining the loss function training model is: Among them, loss(θ) represents the corresponding loss function value, N is the number of samples, |y n | is the length of the nth target keyword sequence, It is the predicted probability distribution using teacher forcing, which is used to ensure that the model matches the target keywords when generating training.

6. A BERT model and keyword extraction method according to claim 5, characterized in that: The process of obtaining the CAIL2018 dataset and inputting each fact description into the trained keyword extraction model to obtain the corresponding keywords includes: Step 1: Take the CAIL 2018 dataset. Here, the maximum length allowed for the model is set to 512 characters. Fact descriptions with a length of less than 512 characters are padded, and fact descriptions with a length of more than 512 characters are truncated from the end of the sentence. Step 2: From step 1, a data set that meets the conditional length will be obtained. This data set is input into the keyword extraction model to obtain the corresponding keyword set.

7. A BERT model and keyword extraction method according to claim 6, characterized in that: The keyword is input into the BERT model to obtain the vector representation of the keyword, and the vector is passed into the cross-attention layer. The result represents the contextual representation of the keyword in the fact description. The process includes: Step 1: For the keyword set D = {x1, x2, ..., x n }, input each word into the pre-trained BERT encoder for encoding, and obtain the corresponding vector representation: {h1,h2,…,h n }=BERT(x1,x2,…,x n ) H = {h1, h2, ..., h n } is the vector sequence of the keyword set, n is the number of keywords; Step 2: Input H into the fully connected layer in the cross-attention layer to obtain U = {u1,u,...,u n }: u i =tanh(W i h i +b i ),i∈[1,n] Step 3: i Use the softmax function to obtain the normalized importance weight α i , which measures the importance of a word: Step 4: Add α i With h i Perform point multiplication and add to get the corresponding hidden state weighted sum, r represents the output of the cross attention layer, that is, the context representation of the corresponding fact description:

8. A BERT model and keyword extraction method according to claim 7, characterized in that: The code is input into the BERT model to obtain the corresponding vector representation, and the vector is passed into the cross-attention layer. The result represents the contextual representation of the keyword in the fact description. The process includes: Step 1: Use the same encoder to encode the character sequence of the code to obtain the corresponding vector H′={h′1, h′2, ..., h′ len }, where len represents the length of the legal article; Step 2: Input H′ into the fully connected layer to obtain U′={u′1,u′2≤,…,u′ len }: u' j =tanh(W′ j h' j +b′ j ),j∈[1,len] Step 3: For u′ j Use the softmax function to obtain the normalized importance weight α′ j , which measures the importance of a word: Step 4: Set α′ j and h′ j Perform point multiplication and add to get the corresponding hidden state weighted sum, c represents the output of the cross attention layer, that is, the context representation of the corresponding code: Through the above four steps, the context representation of each code will be obtained: C = {c1, c2, ..., c l }, l represents the number of articles in the code.

9. A BERT model and keyword extraction method according to claim 8, characterized in that: The results obtained in steps S6 and S7 are matched and calculated to obtain the attention weights of the code provisions on the keywords describing the facts, so as to incorporate the code provisions into the model. Finally, the context description of the code provisions with the highest matching level is input into the softmax layer together with the context representation of the fact description to perform the two classification tasks of crime prediction and law prediction, and the cross entropy loss function is used to train the model. For the regression task of sentence prediction, the above two context representations are input into the linear layer to predict the corresponding results. The process of using the logarithmic distance function as the loss function includes: Step 1: Incorporate the legal code into the model and obtain the attention weight of the legal code to the keywords describing the facts: W ij =f(r i c j ),i∈[1,n],j∈[1,l] Among them, f() represents the inner product function, βi j Indicates the attention weight of the keywords in the code of law on the description of facts; Step 2: Use a scaled dot product attention mechanism to weigh the relative importance of different code provisions: Among them, ρ represents the attention weight of the code provisions, and the weight matrix Step 3: Combine β and ρ through the inner product f() to get the final attention weight γ, and then add γ to r i Perform inner product and perform softmax operation on the result to get q i , and finally all q i Concatenate them into a vector S and use the maximum pooling layer to get the final representation 0: γ=f(β,ρ) q i =softmax(f(γ,r i )) S=[q1,q2,...,q l ] O = maxpooling(S) Step 4: Input the final representation O into the softmax layer to get the final result: Here t and b t They are respectively a trainable matrix and a bias. However, since crime prediction and law prediction are two classification tasks, two softmax operations are performed separately. For sentence prediction, the final representation O of the maximum pooling layer is input into a linear layer and calculated using the ReLU activation function. The formula is as follows: Among them, ω1, ω0, b0, and b1 are all trainable vectors and biases. Each classification task here uses the cross entropy function as the loss function, and the regression task uses the logarithmic distance function as the loss function: For the law prediction task, the cross entropy function is used as the loss function, and the formula is as follows: Among them, |Law| is the number of legal categories, It is the true label of the law. is the model prediction probability distribution; For the crime prediction task, the cross entropy function is used as the loss function, and the formula is as follows: Among them, |Crime| is the number of crime categories, It is the true label of the crime. is the model prediction probability distribution; For the sentence prediction task, the logarithmic distance function is used as the loss function, and the formula is as follows: in, is the sentence predicted by the model, y term is the actual sentence, ∈ is a small value used to avoid errors in logarithmic operations, usually 1e-6. Finally, the loss function is defined as: Loss(θ)=λ ta Loss(t a )+λ tc Loss(t c )+λ to Loss(t p ) Here t a ,t c ,t p Represent three subtasks, λ ta , tc , tp It represents the weight coefficient of each subtask loss, and Loss(θ) represents the total loss function value.

Images