Legal decision prediction model based on judicial knowledge fusion and multi-task dependence shielding

Abstract

The invention discloses a legal decision prediction model based on judicial knowledge fusion and multi-task dependence shielding, which can more effectively complete prediction of three sub-tasks of legal clauses, criminal name commanding and control and criminal period prediction, and relates to application of a natural language processing technology in artificial intelligence in a legal system. According to the judicial knowledge fusion method, two types of judicial knowledge containing rich judicial information, namely legal provisions and command and control explanation, are introduced, a model selects the judicial knowledge most related to the fact of a current case through a multi-head cross-attention mechanism, and noise information brought in the fusion process is eliminated by using the difference of a pair of SoftMax functions. Therefore, the case fact feature representation is enhanced. According to the multi-task dependency shielding method, a prediction result of a front task is judged in a training stage, if the prediction result is correct, task dependency information is transmitted, if the prediction result is incorrect, the task dependency information is shielded, and the wrong task dependency information is skillfully and accurately recognized and shielded.

Description

Technical Field

[0001] The present invention relates to the application of natural language recognition and processing technologies in legal intelligence systems, specifically a legal judgment prediction model based on judicial knowledge fusion and multi-task dependency masking, which can be applied to legal judgment prediction tasks in legal intelligence systems. Background Art

[0002] With the rapid development of information technology and the advent of the big data era, the legal field has also ushered in unprecedented changes. Legal decision prediction, as a key component of this transformation, is gradually becoming a new hot topic in legal practice and research. Legal decision prediction involves predicting and analyzing the likely outcomes of future cases using advanced algorithms and technical means, based on extensive historical case data and relevant legal provisions.

[0003] Currently, the LJP task is defined as a text classification task, and related research and practice focus primarily on text classification methodologies. However, with the continued application of deep learning technology in the legal field, an increasing number of methods have been proposed, demonstrating strong feasibility. However, it cannot be ignored that, in practical applications, the LJP task faces numerous complex and challenging challenges. Specifically, the case fact descriptions in the LJP task are often complex and diverse, and the logical relationships between the case elements they cover are complex, making it difficult to accurately extract key information and use it for prediction. Furthermore, legal provision prediction or charge prediction tasks are prone to confusion, as some legal provisions or charges share certain similarities in terms of wording and scope of application, which can easily lead to biased model judgments. Furthermore, formalizing the sentence prediction task as a text classification task results in large variations in the prediction interval.

[0004] On the one hand, most early LJP research focused solely on mining case element information contained in case facts, then making corresponding predictions for each subtask based on the semantic information of these case elements. However, most of these studies failed to fully consider the rich judicial knowledge contained in legal provisions and the interpretation of charges. On the other hand, multi-task learning has become a widely adopted standard modeling approach in LJP research. In actual legal decision-making scenarios, human judges follow a specific decision-making logic, first accurately determining the corresponding legal provisions based on the case facts, then clarifying the charges based on the determined legal provisions, and finally determining the sentence based on the legal provisions and the charges. Summary of the Invention

[0005] The present invention discloses a legal judgment prediction model based on judicial knowledge fusion and multi-task dependency masking, which is characterized by comprising the following steps:

[0006] S1. Use the LERT pre-trained language model to extract features from two types of external judicial knowledge: legal provisions and accusation explanations, to obtain a judicial knowledge feature representation.

[0007] S2. Use the LERT pre-trained language model to extract case facts from judgment samples to obtain a case fact feature representation. Multi-head cross-attention differentials are then used to integrate the judicial knowledge feature representation into the case fact feature representation. A residual connection operation is then used to prevent feature degradation, resulting in a judicial knowledge-enhanced case fact feature representation.

[0008] S3. The case fact feature representation augmented with judicial knowledge is input into an LSTM long short-term memory network consisting of three time steps. The network then predicts the legal clauses, charges, and sentence in the legal judgment. The network then judges the prediction results of the previous task. If the prediction is correct, the task-dependent information is transferred; if it is incorrect, the task-dependent information is masked. A multi-task decoder consisting of fully connected layers is then used to predict the probability of the legal clause classification, the charge classification, and the sentence classification for the judgment sample. (The legal clause is the abbreviation of the legal clause, and the charge is the abbreviation of the charge.)

[0009] S4. Use the cross-entropy loss function to calculate the losses of the three subtasks of legal clause, charge, and sentence prediction, and add them together to obtain the final prediction loss for model training.

[0010] The LERT pre-trained language model refers to a language information enhanced pre-training model LERT (Linguistically-motivated bidirectional Encoder Representation from Transformer) proposed by the Harbin Institute of Technology iFlytek Joint Laboratory in the paper “Cui Y, Che W, Liu T, Qin B Yang Z (2021) Pre-Training With Whole Word Masking for Chinese BERT. In: IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp 3504-3514”;

[0011] The LSTM long short-term memory network was proposed by Hochreiter and Schmidhuber in the paper “Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735-1780”;

[0012] Furthermore, the step S1 specifically includes:

[0013] S1.1 Extract the definition text of each legal clause tag in the Criminal Law of the People's Republic of China to obtain the corresponding legal clause, forming the legal clause set A = {A1, A2, A3..., A m}, m is the number of legal clause labels in the corpus, A m represents the text of the mth legal provision in A, and then uses the following formula to calculate the i-th legal provision A in A i The eigenvector of

[0014]

[0015] Among them, Tokenizer(·) represents the word segmenter of the LERT pre-trained language model, A is obtained after being processed by the Tokenizer i Token subword sequence, n A Indicates the fixed length of the legal provision, the fixed length of the legal provision n A It means when The length is less than n A When passing A i Pad the end with 0 to reach n A ,when The length is greater than n A When its tail is greater than n A The part is cut off, LERT(·) represents the LERT pre-trained language model; Indicates A i The hidden state sequence in the LERT model, d is the dimension of the hidden state in the LERT model, and R represents a real number set;

[0016] S1.2 Repeat step S1.1 and perform feature extraction on all legal provisions in legal provision set A to obtain:

[0017]

[0018] Among them, v A Represents the eigenvector combination of the legal clause set A, Indicates A in A m The eigenvector of

[0019] 1.3 Extract the explanation text of each crime charge label in the corpus from the Criminal Law of the People's Republic of China to obtain the corresponding accusation explanation text, forming the accusation explanation set C of the corpus = {C1, C2, C3, ..., C k}, k is the number of crime accusation labels in the corpus, C k represents the text of the kth accusation explanation in C, and then uses the following formula to calculate the i-th accusation explanation C in C i The eigenvector of

[0020]

[0021] in, C is obtained after being processed by the Tokenizer i Token subword sequence, n C The fixed length of the token subword sequence representing the accusation interpretation, Indicates C i Hidden state sequence in the LERT model;

[0022] S1.4 Repeat step S1.3 and perform feature extraction on all accusation explanations in the accusation explanation set C to obtain:

[0023]

[0024] Among them, v C represents the eigenvector combination of the accusation explanation set C, Indicates C in C k The eigenvector of

[0025] S1.5 uses convolutional neural network (CNN) to process v one by one A The feature vectors of legal provisions in are used to further mine the main legal knowledge in the legal provisions:

[0026]

[0027] Among them, Conv(·) represents the convolution operation, Represents the feature vector of the i-th legal provision in A The convolution kernel of the processed convolutional neural network has a size of 3×3, an input channel of d, and an output channel of d / 3. is the bias of the convolution, Express The processing results;

[0028] S1.6 Let i be equal to 1 to m, repeat step S1.5, andA All the feature vectors in are processed and the processing results are spliced together to obtain the main legal knowledge features

[0029]

[0030] Among them, concat(·) represents the concatenation operation;

[0031] S1.7 uses convolutional neural network (CNN) to process v one by one C The accusation explanation feature vector in is used to further mine the main accusation knowledge in the accusation explanation:

[0032]

[0033] in, Represents the feature vector used to explain the i-th accusation in C The convolution kernel of the processed convolutional neural network has a size of 3×3, an input channel of d, and an output channel of d / 3. is the bias of the convolution;

[0034] S1.8 Let i be equal to 1 to k, repeat step S1.7, and C All the accusation explanation feature vectors in are processed and the processing results are spliced together to obtain the main accusation knowledge features

[0035]

[0036] 1.9 pairs and Apply linear transformation to map the dimension back to d and obtain the main legal knowledge features of dimension d and main accusation knowledge characteristics

[0037]

[0038] Among them, W A ∈R (d / 3×m)×d 、b A ∈R d They are respectively processed The linear transformation weight matrix and bias, W C ∈R (d / 3×k)×d 、b c ∈R d They are respectively processed The weight matrix and bias of the linear transformation, Relu(·) represents the ReLU activation function;

[0039] 1.10 will and The sequence lengths are uniformly expanded to length n by filling 0 s , and then add them point by point to obtain the judicial knowledge feature representation v containing legal provisions and accusation explanations s :

[0040]

[0041] Among them, n S Indicates v S The sequence length is n S >n A With n S >n C ;

[0042] Furthermore, the step S2 specifically includes:

[0043] S2.1 Use the LERT pre-trained language model to extract features from the case fact text f of the judgment sample and obtain the case fact feature representation v f :

[0044]

[0045] in, is the token subword sequence of f obtained after being processed by the Tokenizer, n f Indicates the length of the token subword sequence of f;

[0046] S2.2: Applying cross-attention differentials to integrate judicial knowledge feature representation into case fact feature representation:

[0047] First, a linear layer is used to represent the fact feature v f and judicial knowledge feature representation v S Perform a linear transformation to prepare parameters for two differential scaled dot product attentions, where the fact feature representation v f The linear transformation of is used as the attention query Q, and the judicial knowledge feature representation v S The linear transformation of is used as the key K and value V of attention. The specific calculation process is as follows:

[0048]

[0049] in, are the linear transformation matrices of the query Q and key K of the first attention, are the linear transformation matrices of the query Q and key K of the second attention, W V ∈R d×d is the linear transformation matrix of the value V shared by the two attentions, Based on vf Linearly transform the query Q parameters of the first and second attentions, Based on v S The key K parameters of the first and second attention of the linear transformation, V S Based on v S The shared value parameter V of the two attentions of the linear transformation;

[0050] Then, calculate the difference between the two attentions and get the cross-attention difference result DiffAttn:

[0051]

[0052] Where softmax(·) represents the Softmax activation function, T represents the matrix transpose, λ is a hyperparameter, and λ=0.2;

[0053] S2.3 uses multiple heads to enhance the cross-attention difference calculation in step S2.2, learns relevant judicial knowledge from different dimensions, and obtains multi-head cross-attention difference results

[0054]

[0055] in, Indicates the Q and K parameters for calculating the attention value of the i-th head, W O ∈R d×d is a learnable weight matrix, Indicates the attention value of the i-th head, DiffAttn i represents the cross-attention difference result of the i-th head, MultiHeadAttn(·) represents the scaled dot product multi-head attention, and LN(·) represents the linear normalization processing;

[0056] S2.4 In order to reduce the loss of case fact information due to network depth during the judicial knowledge fusion process, a residual connection is performed after the multi-head cross-attention difference calculation, which is calculated as follows:

[0057]

[0058] in, is the case fact feature representation enhanced by judicial knowledge, Res(·) represents the residual connection processing;

[0059] Furthermore, the step S3 specifically includes:

[0060] S3.1 Representing case fact features enhanced by judicial knowledge Input into an LSTM long short-term memory network consisting of three time steps, respectively for the legal clause prediction subtask in the legal judgmentl , accusation prediction subtask c and sentence prediction subtask t Make predictions and define the following masked task dependencies for these three subtasks: task l ≥task c 、task l ≥task t and task c ≥task t , where the symbol “≥” represents a masked task dependency, which means that during the training phase, the features of the predecessor task are transferred to the subsequent task only when the prediction of the predecessor task is true, otherwise they are not transferred; if the features of the predecessor task are transferred successfully, the subsequent task depends on the features of the predecessor task for prediction;

[0061] S3.2 Use the first time step in the LSTM network to predict the law task l Perform calculations and predictions:

[0062] First, calculate the legal clause prediction task l The decoded hidden state of :

[0063]

[0064] Among them, LSTMCell1(·) represents the first time step in the LSTM network, Represents the initial hidden state and initial memory unit of LSTMCell1, H l ∈R d Represents the legal clause prediction task obtained after processing by LSTMCell1 l The decoded hidden state, z l Represents the value of the memory cell after processing by LSTMCell1;

[0065] Then, a fully connected layer that performs the Softmax function is used to calculate the law prediction task task l The prediction results are:

[0066]

[0067] Among them, W l ∈R m×d 、b l ∈R m They are tasks l The trainable weight matrices and biases in the task, is the probability distribution on the legal article labels, is the predicted label of the legal clause prediction task, Express request The legal label corresponding to the element with the highest probability, i1 represents The element subscript in , express The i1th element in ;

[0068] Finally, according to Whether the prediction is correct determines whether the task l Task information is updated for subsequent subtasks:

[0069]

[0070] Among them, l t To predict the sample in task l The true label in the task, when When Mask l The value is True, otherwise it is False, Mask l For task l The mask for task information to be passed back;

[0071] S3.3 Use the second time step in the LSTM network to predict the accusation task c Perform calculations and predictions:

[0072] First, Mask l Applied to the following calculations to achieve the task l Filtering of incorrect task dependency information in tasks:

[0073]

[0074] Among them, W l,c ∈R d×d 、b l，c ∈R d Through task l Task information update task c The initial hidden state of the task and the weight matrix and bias of the memory unit, Represents Mask c The initial value of the hidden state and memory unit in the task, when Mask l =True, Why use formula (30) to calculate, otherwise

[0075] Then, use the following formula to calculate the charge prediction task task c The decoded hidden state of :

[0076]

[0077] Among them, LSTMCell2(·) represents the second time step in the LSTM network, H c ∈R d Indicates the charge prediction task obtained after processing by LSTMCell2 c The decoded hidden state, Z c Represents the value of the memory unit after LSTMCell2 processing;

[0078] Then, a fully connected layer that performs a Softmax function is used to calculate the charge prediction task. c The prediction results are:

[0079]

[0080] Among them, W c ∈R k×d 、b l ∈R k It's a task c The trainable weight matrices and biases in the task, is the probability distribution over the accusation labels, is the predicted label of the accusation prediction task, i2 represents The element subscript in , express The i2th element in ;

[0081] Finally, according to Whether the prediction is correct determines whether the task c Task information is updated for subsequent subtasks:

[0082]

[0083] when When, that is, task c The prediction result of the accusation prediction task is correct, Mask c The value is True, otherwise it is False, the c t Indicates that the predicted sample is in task c The actual label in the task, Mask c For task c The mask for task information to be passed back;

[0084] S3.4 Use the third time step in the LSTM network to predict the sentence task t Perform calculations and predictions:

[0085] First, Mask l and Mask c Applied to the following calculations to achieve the task lLegal clause prediction tasks and tasks c Filtering of incorrect task-dependent information in the accusation prediction task:

[0086]

[0087] Among them, W l, t∈R d×d 、b l,t ∈R d Through task l Task information update task t The initial hidden state and memory unit weight matrix and bias for the sentence prediction task, W c,t ∈R d×d 、b c,t ∈R d Through task c Task information update task t The initial hidden state and memory unit weight matrix and bias for the sentence prediction task, task t The initial values of the hidden states and memory units in the sentence prediction task are only valid when Mask l =True or Mask c = True, the corresponding Information is involved in the calculation of formula (35). If Mask l = False and Mask c = False, then

[0088] Then, use the following formula to calculate task t Decoded hidden state for the prison sentence prediction task:

[0089]

[0090] Among them, LSTMCell3(·) represents the third time step in the LSTM network, H t ∈R d Indicates the task obtained after LSTMCell3 processing t Decoded hidden state for sentence prediction task, Z t Indicates the value of the memory cell after LSTMCell3 processing;

[0091] Finally, a fully connected layer that performs a Softmax function is used to calculate the sentence prediction task t The prediction results are:

[0092]

[0093] Among them, Wt ≥R j×d 、b t ∈R j It's a task t Trainable weight matrices and biases for the sentence prediction task, is the probability distribution over sentence labels, j is the number of sentence labels in the corpus, is the predicted label of the sentence prediction task, i3 represents The element subscript in , express The i3th element in .

[0094] Furthermore, the step S4 specifically includes:

[0095] This step calculates the cross entropy loss for each subtask and uses the sum of the losses of all subtasks as the final overall loss to backpropagate the optimization model:

[0096]

[0097] Among them, |Y X | represents the number of labels for each subtask X, Indicates the final loss.

[0098] The present invention has the following advantages:

[0099] (1) Two types of judicial knowledge containing rich judicial information, namely legal provisions and accusation explanations, are introduced. The multi-head cross-attention differential mechanism is used to enable the model to select the judicial knowledge most relevant to the current case facts for integration, and a pair of SoftMax functions is used to eliminate the noise information brought about by the integration process, thereby enhancing the representation of case fact features.

[0100] (2) A multi-task research approach is adopted to establish a multi-task research system with task dependency masking for the three subtasks. During the training phase, the prediction results of the previous task are judged. If the prediction is correct, the task dependency information is transmitted; otherwise, the task dependency information is masked. This method cleverly identifies and masks the incorrect task dependency information. BRIEF DESCRIPTION OF THE DRAWINGS

[0101] Figure 1 A legal judgment prediction model based on judicial knowledge fusion and multi-task dependency masking in an embodiment;

[0102] Figure 2 It is the multi-head cross-attention differential judicial knowledge fusion mechanism in the embodiment;

[0103] Figure 3 Examples of case facts in the embodiments that are enhanced by judicial knowledge. DETAILED DESCRIPTION

[0104] The present invention will be further described below with reference to specific examples, but the protection scope of the present invention is not limited to the following examples.

[0105] The present invention discloses a legal judgment prediction model based on judicial knowledge fusion and multi-task dependency masking, which is characterized by comprising the following steps:

[0106] S1. Use the LERT pre-trained language model to extract features from two types of external judicial knowledge: legal provisions and accusation explanations, to obtain a judicial knowledge feature representation.

[0107] S2. Use the LERT pre-trained language model to extract case facts from judgment samples to obtain a case fact feature representation. Multi-head cross-attention differentials are then used to integrate the judicial knowledge feature representation into the case fact feature representation. A residual connection operation is then used to prevent feature degradation, resulting in a judicial knowledge-enhanced case fact feature representation.

[0108] S3. The case fact feature representation augmented with judicial knowledge is input into an LSTM long short-term memory network consisting of three time steps. The network then predicts the legal clauses, charges, and sentence in the legal judgment. The network then judges the prediction results of the previous task. If the prediction is correct, the task-dependent information is transferred; if it is incorrect, the task-dependent information is masked. A multi-task decoder consisting of fully connected layers is then used to predict the probability of the legal clause classification, the charge classification, and the sentence classification for the judgment sample. (The legal clause is the abbreviation of the legal clause, and the charge is the abbreviation of the charge.)

[0109] S4. Use the cross-entropy loss function to calculate the losses of the three subtasks of legal clause, charge, and sentence prediction, and add them together to obtain the final prediction loss for model training.

[0110] The LERT pre-trained language model refers to a language information enhanced pre-training model LERT (Linguistically-motivated bidirectional Encoder Representation from Transformer) proposed by the Harbin Institute of Technology iFlytek Joint Laboratory in the paper “Cui Y, Che W, Liu T, Qin B Yang Z (2021) Pre-Training With Whole Word Masking for Chinese BERT. In: IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp 3504-3514”;

[0111] The LSTM long short-term memory network was proposed by Hochreiter and Schmidhuber in the paper “Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735-1780”;

[0112] Furthermore, the step S1 specifically includes:

[0113] S1.1 Extract the definition text of each legal clause tag in the Criminal Law of the People's Republic of China to obtain the corresponding legal clause, forming the legal clause set A = {A1, A2, A3..., A m}, m is the number of legal clause labels in the corpus, A m represents the text of the mth legal provision in A, and then uses the following formula to calculate the i-th legal provision A in A i The eigenvector of

[0114]

[0115] Among them, Tokenizer(·) represents the word segmenter of the LERT pre-trained language model, A is obtained after being processed by the Tokenizer i Token subword sequence, n A Indicates the fixed length of the legal provision, the fixed length of the legal provision n A It means when The length is less than n A When passing A i Pad the end with 0 to reach n A ,when The length is greater than n A When its tail is greater than n A The part is cut off, LERT(·) represents the LERT pre-trained language model; Indicates A i The hidden state sequence in the LERT model, d is the dimension of the hidden state in the LERT model, and R represents a real number set;

[0116] S1.2 Repeat step S1.1 and perform feature extraction on all legal provisions in legal provision set A to obtain:

[0117]

[0118] Among them, v ARepresents the eigenvector combination of the legal clause set A, Indicates A in A m The eigenvector of

[0119] 1.3 Extract the explanation text of each crime charge label in the corpus from the Criminal Law of the People's Republic of China to obtain the corresponding accusation explanation text, forming the accusation explanation set C of the corpus = {C1, C2, C3, ..., C k}, k is the number of crime accusation labels in the corpus, C k represents the text of the kth accusation explanation in C, and then uses the following formula to calculate the i-th accusation explanation C in C i The eigenvector of

[0120]

[0121] in, C is obtained after being processed by the Tokenizer i Token subword sequence, n C The fixed length of the token subword sequence representing the accusation interpretation, Indicates C i Hidden state sequence in the LERT model;

[0122] S1.4 Repeat step S1.3 and perform feature extraction on all accusation explanations in the accusation explanation set C to obtain:

[0123]

[0124] Among them, v C represents the eigenvector combination of the accusation explanation set C, Indicates C in C k The eigenvector of

[0125] S1.5 uses convolutional neural network (CNN) to process v one by one A The feature vectors of legal provisions in are used to further mine the main legal knowledge in the legal provisions:

[0126]

[0127] Among them, Conv(·) represents the convolution operation, Represents the feature vector of the i-th legal provision in A The convolution kernel of the processed convolutional neural network has a size of 3×3, an input channel of d, and an output channel of d / 3. is the bias of the convolution, Express The processing results;

[0128] S1.6 Let i be equal to 1 to m, repeat step S1.5, and A All the feature vectors in are processed and the processing results are spliced together to obtain the main legal knowledge features

[0129]

[0130] Among them, concat(·) represents the concatenation operation;

[0131] S1.7 uses convolutional neural network (CNN) to process v one by one C The accusation explanation feature vector in is used to further mine the main accusation knowledge in the accusation explanation:

[0132]

[0133] in, Represents the feature vector used to explain the i-th accusation in C The convolution kernel of the processed convolutional neural network has a size of 3×3, an input channel of d, and an output channel of d / 3. is the bias of the convolution;

[0134] S1.8 Let i be equal to 1 to k, repeat step S1.7, and C All the accusation explanation feature vectors in are processed and the processing results are spliced together to obtain the main accusation knowledge features

[0135]

[0136] 1.9 pairs and Apply linear transformation to map the dimension back to d and obtain the main legal knowledge features of dimension d and main accusation knowledge characteristics

[0137]

[0138] Among them, W A ∈R (d / 3×m)×d 、b A ∈R d They are respectively processed The linear transformation weight matrix and bias, W C ∈R (d / 3×k)×d 、b C ∈R d They are respectively processed The weight matrix and bias of the linear transformation, Relu(·) represents the ReLU activation function;

[0139] 1.10 will and The sequence lengths are uniformly expanded to length n by filling 0 S , and then add them point by point to obtain the judicial knowledge feature representation v containing legal provisions and accusation explanations S :

[0140]

[0141] Among them, n S Indicates v S The sequence length is n S >n A With n S >n C ;

[0142] Furthermore, the step S2 specifically includes:

[0143] S2.1 Use the LERT pre-trained language model to extract features from the case fact text f of the judgment sample and obtain the case fact feature representation v f :

[0144]

[0145] in, is the token subword sequence of f obtained after being processed by the Tokenizer, n f Indicates the length of the token subword sequence of f;

[0146] S2.2: Applying cross-attention differentials to integrate judicial knowledge feature representation into case fact feature representation:

[0147] First, a linear layer is used to represent the fact feature v f and judicial knowledge feature representation v S Perform a linear transformation to prepare parameters for two differential scaled dot product attentions, where the fact feature representation v f The linear transformation of is used as the attention query Q, and the judicial knowledge feature representation v S The linear transformation of is used as the key K and value V of attention. The specific calculation process is as follows:

[0148]

[0149] in, are the linear transformation matrices of the query Q and key K of the first attention, are the linear transformation matrices of the query Q and key K of the second attention, W V ∈Rd×d is the linear transformation matrix of the value V shared by the two attentions, Based on v f Linearly transform the query Q parameters of the first and second attentions, Based on v S The key K parameters of the first and second attention of the linear transformation, V S Based on v S The shared value parameter V of the two attentions of the linear transformation;

[0150] Then, calculate the difference between the two attentions and get the cross-attention difference result DiffAttn:

[0151]

[0152] Where softmax(·) represents the Softmax activation function, T represents the matrix transpose, λ is a hyperparameter, and λ=0.2;

[0153] S2.3 uses multiple heads to enhance the cross-attention difference calculation in step S2.2, learns relevant judicial knowledge from different dimensions, and obtains multi-head cross-attention difference results

[0154]

[0155]

[0156] in, Indicates the Q and K parameters for calculating the attention value of the i-th head, W O ∈R d×d is a learnable weight matrix, Indicates the attention value of the i-th head, DiffAttn i represents the cross-attention difference result of the i-th head, MultiHeadAttn(·) represents the scaled dot product multi-head attention, and LN(·) represents the linear normalization processing;

[0157] S2.4 In order to reduce the loss of case fact information due to network depth during the judicial knowledge fusion process, a residual connection is performed after the multi-head cross-attention difference calculation, which is calculated as follows:

[0158]

[0159] in, is the case fact feature representation enhanced by judicial knowledge, Res(·) represents the residual connection processing;

[0160] Furthermore, the step S3 specifically includes:

[0161] S3.1 Representing case fact features enhanced by judicial knowledge Input into an LSTM long short-term memory network consisting of three time steps, respectively for the legal clause prediction subtask in the legal judgment l , accusation prediction subtask c and sentence prediction subtask t Make predictions and define the following masked task dependencies for these three subtasks: task l ≥task c 、task l ≥task t and task c ≥task t , where the symbol “≥” represents a masked task dependency, which means that during the training phase, the features of the predecessor task are transferred to the subsequent task only when the prediction of the predecessor task is true, otherwise they are not transferred; if the features of the predecessor task are transferred successfully, the subsequent task depends on the features of the predecessor task for prediction;

[0162] S3.2 Use the first time step in the LSTM network to predict the law task l Perform calculations and predictions:

[0163] First, calculate the legal clause prediction task l The decoded hidden state of :

[0164]

[0165] Among them, LSTMCell1(·) represents the first time step in the LSTM network, Represents the initial hidden state and initial memory unit of LSTMCell1, H l ∈R d Represents the legal clause prediction task obtained after processing by LSTMCell1 l The decoded hidden state, Z l Represents the value of the memory cell after processing by LSTMCell1;

[0166] Then, a fully connected layer that performs the Softmax function is used to calculate the law prediction task task l The prediction results are:

[0167]

[0168] Among them, W l ∈R m×d 、b l ∈R m They are tasks lThe trainable weight matrices and biases in the task, is the probability distribution on the legal article labels, is the predicted label of the legal clause prediction task, Express request The legal label corresponding to the element with the highest probability, i1 represents The element subscript in , express The i1th element in ;

[0169] Finally, according to Whether the prediction is correct determines whether the task l Task information is updated for subsequent subtasks:

[0170]

[0171] Among them, l t To predict the sample in task l The true label in the task, when When Mask l The value is True, otherwise it is False, Mask l For task l The mask for task information to be passed back;

[0172] S3.3 Use the second time step in the LSTM network to predict the accusation task c Perform calculations and predictions:

[0173] First, Mask l Applied to the following calculations to achieve the task l Filtering of incorrect task dependency information in tasks:

[0174]

[0175] Among them, W l,c ∈R d×d 、b l,c ∈R d Through task l Task information update task c The initial hidden state of the task and the weight matrix and bias of the memory unit, Respectively represent tasks c The initial value of the hidden state and memory unit in the task, when Mask l =Ture, Why use formula (30) to calculate, otherwise

[0176] Then, use the following formula to calculate the charge prediction task taskc The decoded hidden state of :

[0177]

[0178] Among them, LSTMCell2(·) represents the second time step in the LSTM network, H c ∈R d Indicates the charge prediction task obtained after processing by LSTMCell2 c The decoded hidden state, Z c Represents the value of the memory unit after LSTMCell2 processing;

[0179] Then, a fully connected layer that performs a Softmax function is used to calculate the charge prediction task. c The prediction results are:

[0180]

[0181] Among them, W c ∈R k×d 、b l ∈R k It's a task c The trainable weight matrices and biases in the task, is the probability distribution over the accusation labels, is the predicted label of the accusation prediction task, i2 represents The element subscript in , express The i2th element in ;

[0182] Finally, according to Whether the prediction is correct determines whether the task c Task information is updated for subsequent subtasks:

[0183]

[0184] when When, that is, task c The prediction result of the accusation prediction task is correct, Mask c The value is True, otherwise it is False, the c t Indicates that the predicted sample is in task c The actual label in the task, Mask c For task c The mask for task information to be passed back;

[0185] S3.4 Use the third time step in the LSTM network to predict the sentence task t Perform calculations and predictions:

[0186] First, Mask l and Mask c Applied to the following calculations to achieve the task l Legal clause prediction tasks and tasks c Filtering of incorrect task-dependent information in the accusation prediction task:

[0187]

[0188] Among them, W l，t ∈R d×d 、b l，t ∈R d Through task l Task information update task t The initial hidden state and memory unit weight matrix and bias for the sentence prediction task, W c，t ∈R d×d 、b c，t ∈R d Through task c Task information update task t The initial hidden state and memory unit weight matrix and bias for the sentence prediction task, task t The initial values of the hidden states and memory units in the sentence prediction task are only valid when Mask l =True or Mask c = True, the corresponding Information is involved in the calculation of formula (35). If Mask l = False and Mask c = False, then

[0189] Then, use the following formula to calculate task t Decoded hidden state for the prison sentence prediction task:

[0190]

[0191] Among them, LSTMCell3(·) represents the third time step in the LSTM network, H t ∈R d Indicates the task obtained after LSTMCell3 processing t Decoded hidden state for sentence prediction task, Z t Indicates the value of the memory cell after LSTMCell3 processing;

[0192] Finally, a fully connected layer that performs a Softmax function is used to calculate the sentence prediction task.t The prediction results are:

[0193]

[0194] Among them, W t ∈R j×d 、b t ∈R j It's a task t Trainable weight matrices and biases for the sentence prediction task, is the probability distribution over sentence labels, j is the number of sentence labels in the corpus, is the predicted label of the sentence prediction task, i3 represents The element subscript in , express The i3th element in .

[0195] Furthermore, the step S4 specifically includes:

[0196] This step calculates the cross entropy loss for each subtask and uses the sum of the losses of all subtasks as the final overall loss to backpropagate the optimization model:

[0197]

[0198] Among them, |Y X | represents the number of labels for each subtask X, Indicates the final loss.

[0199] Application Examples

[0200] 1. Instance environment

[0201] The base version of the LERT pre-trained model is used as the encoder of the instance, and n in the formula A =256, n C =256·n S =256. The hyperparameter settings of the model are shown in Table 1

[0202] Table 1: Hyperparameters of the examples

[0203]

[0204] 2. Dataset

[0205] This example uses the CAIL_small and CAIL_big datasets, available for download at: https: / / cail.oss-cn-qingdao.aliyuncs.com / CAIL2018_ALL_DATA.zip. These datasets are from the China AI Legal Challenge Legal Judgment Prediction Task and are publicly available. The datasets are derived from judgments issued by the Supreme People's Court of the People's Republic of China, including factual statements, legal provisions, charges, and sentences. This example focuses solely on cases involving a single legal provision and a single charge. Furthermore, cases with missing labels and cases where the legal provision or charge appears infrequently are filtered out. For sentence length, 11 intervals are set for classification. Since CAIL_big does not provide a validation set, the processed training dataset is split 9:1 for training and validation purposes. Furthermore, this example filters out legal provision and charge labels with a frequency of less than 100.

[0206] 3. Comparison Method

[0207] TOPJUDGE[1] completes the prediction task by constructing a directed acyclic graph using the unidirectional topological dependencies between the three subtasks of LJP.

[0208] MPBFN-WCA[2] builds a multi-view bidirectional feedback network based on the topological structure between subtasks and uses word collocation attention mechanism to improve prediction performance.

[0209] LANDAN[3] uses an end-to-end LJP framework to distinguish confusing legal terms by building a self-learning graph attention network.

[0210] EPM[4] adds a penalty term to the loss function, uses the relationship between the three subtasks to extract event features, and uses event information to assist in predicting and judging results.

[0211] References:

[0212] [1]Zhong H, Guo Z, Tu C, Xiao C, Liu Z, Sun M (2018) Legal JudgmentPrediction via Topological Learning. In Proceedings of the 2018Conference on Empirical Methods in Natural Language Processing, pp 3540-3549, Brussels, Belgium. Association for Computational Linguistics.

[0213] [2] Yang W, Jia W, Zhou X, Luo Y (2019) Legal judgment prediction via multi-perspective bi-feedback network. In: Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI'19). AAAIPress, pp 4085-4091.

[0214] [3]Xu N, Wang P, Chen L, Pan L, Wang

[0215] [4] Feng Y, Li C, Vincent Ng (2022) Legal Judgment Prediction via EventExtraction with Constraints. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp 648-664, Dublin, Ireland. Association for Computational Linguistics.

[0216] 4. Example comparison results

[0217] Table 2: Comparison results of CAIL_small dataset instances

[0218]

[0219] Table 3: Comparison results of CAIL_big dataset instances

[0220]

[0221] The results in Tables 2 and 3 show that the method proposed in this example is significantly better than various baseline models in terms of the F1 value of legal clause prediction, charge prediction F1 value, and sentence prediction F1 value. This fully proves that the legal judgment prediction model based on judicial knowledge fusion and multi-task dependency masking proposed in this invention is feasible and excellent.

Claims

1. A legal judgment prediction model based on judicial knowledge fusion and multi-task dependency masking, characterized by The following steps are involved: S1. Use the LERT pre-trained language model to extract features from two types of external judicial knowledge: legal provisions and accusation explanations, to obtain a judicial knowledge feature representation. S2. Use the LERT pre-trained language model to extract case facts from judgment samples to obtain a case fact feature representation. Multi-head cross-attention differentials are then used to integrate the judicial knowledge feature representation into the case fact feature representation. A residual connection operation is then used to prevent feature degradation, resulting in a judicial knowledge-enhanced case fact feature representation. S3. The case fact feature representation augmented with judicial knowledge is input into an LSTM long short-term memory network consisting of three time steps. The network then predicts the legal clauses, charges, and sentence in the legal judgment. The network then judges the prediction results of the previous task. If the prediction is correct, the task-dependent information is transferred; if it is incorrect, the task-dependent information is masked. A multi-task decoder consisting of fully connected layers is then used to predict the probability of the legal clause classification, the charge classification, and the sentence classification for the judgment sample. (The legal clause is the abbreviation of the legal clause, and the charge is the abbreviation of the charge.) S4. Use the cross-entropy loss function to calculate the losses of the three subtasks of legal clause, charge, and sentence prediction, and add them together to obtain the final prediction loss for model training. The step S1 specifically includes: S1.1 Extract the definition text of each legal clause tag in the Criminal Law of the People's Republic of China to obtain the corresponding legal clause, forming the legal clause set A = {A1, A2, A3..., A m }, m is the number of legal clause labels in the corpus, A m represents the text of the mth legal provision in A, and then uses the following formula to calculate the i-th legal provision A in A i The eigenvector of Among them, Tokenizer(·) represents the word segmenter of the LERT pre-trained language model, A is obtained after being processed by the Tokenizer i Token subword sequence, n A Indicates the fixed length of the legal provision, the fixed length of the legal provision n A It means when The length is less than n A When passing A i Pad the end with 0 to reach n A ,when The length is greater than n A When its tail is greater than n A The part is cut off, LERT(·) represents the LERT pre-trained language model; Indicates A i The hidden state sequence in the LERT model, d is the dimension of the hidden state in the LERT model, and R represents a real number set; S1.2 Repeat step S1.1 and perform feature extraction on all legal provisions in legal provision set A to obtain: Among them, v A Represents the eigenvector combination of the set of legal provisions A, Indicates A in A m The eigenvector of 1.3 Extract the explanation text of each crime charge label in the corpus from the Criminal Law of the People's Republic of China to obtain the corresponding accusation explanation text, forming the accusation explanation set C of the corpus = {C1, C2, C3, ..., C k }, k is the number of crime charge labels in the corpus, C k represents the text of the kth accusation explanation in C, and then uses the following formula to calculate the i-th accusation explanation C in C i The eigenvector of in, C is obtained after being processed by the Tokenizer i Token subword sequence, n C The fixed length of the token subword sequence representing the accusation interpretation, Indicates C i Hidden state sequence in the LERT model; S1.4 Repeat step S1.3 and perform feature extraction on all accusation explanations in the accusation explanation set C to obtain: Among them, v C represents the eigenvector combination of the accusation explanation set C, Indicates C in C k The eigenvector of S1.5 uses convolutional neural network CNN to process v one by one A The feature vectors of legal provisions in are used to further mine the main legal knowledge in the legal provisions: Among them, Conv(·) represents the convolution operation, Represents the feature vector of the i-th legal provision in A The convolution kernel of the processed convolutional neural network has a size of 3×3, an input channel of d, and an output channel of d / 3. is the bias of the convolution, Express The processing results; S1.6 Let i be equal to 1 to m, repeat step S1.5, and A All the feature vectors in are processed and the processing results are spliced together to obtain the main legal knowledge features Among them, concat(·) represents the concatenation operation; S1.7 uses convolutional neural network (CNN) to process v one by one C The accusation explanation feature vector in is used to further mine the main accusation knowledge in the accusation explanation: in, Represents the feature vector used to explain the i-th accusation in C The convolution kernel of the processed convolutional neural network has a size of 3×3, an input channel of d, and an output channel of d / 3. is the bias of the convolution; S1.8 Let i be equal to 1 to k, repeat step S1.7, and C All the accusation explanation feature vectors in are processed and the processing results are spliced together to obtain the main accusation knowledge features 1.9 pairs and Apply linear transformation to map the dimension back to d and obtain the main legal knowledge features of dimension d and main accusation knowledge characteristics in, b A ∈R d They are respectively processed The weight matrix and bias of the linear transformation, b C ∈R d They are respectively processed The weight matrix and bias of the linear transformation, Relu(·) represents the ReLU activation function; 1.10 will and The sequence lengths are uniformly expanded to length n by filling 0 s , and then add them point by point to obtain the judicial knowledge feature representation v containing legal provisions and accusation explanations s : Among them, n s Indicates v s The sequence length is n S >n A With n S >n C ; The step S2 specifically includes: S2.1 Use the LERT pre-trained language model to extract features from the case fact text f of the judgment sample and obtain the case fact feature representation v f : in, is the token subword sequence of f obtained after being processed by the Tokenizer, n f Indicates the length of the token subword sequence of f; S2.2: Applying cross-attention differentials to integrate judicial knowledge feature representation into case fact feature representation: First, a linear layer is used to represent the fact feature v f and judicial knowledge feature representation v S Perform a linear transformation to prepare parameters for two differential scaled dot product attentions, where the fact feature representation v f The linear transformation of is used as the attention query Q, and the judicial knowledge feature representation v S The linear transformation of is used as the key K and value V of attention. The specific calculation process is as follows: in, are the linear transformation matrices of the query Q and key K of the first attention, are the linear transformation matrices of the query Q and key K of the second attention, W V ∈R d×d is the linear transformation matrix of the value V shared by the two attentions, Based on v f Linearly transform the query Q parameters of the first and second attentions, Based on v S The key K parameters of the first and second attention of the linear transformation, V S Based on v S The shared value parameter V of the two attentions of the linear transformation; Then, calculate the difference between the two attentions and get the cross-attention difference result DiffAttn: Where softmax(·) represents the Softmax activation function, T represents the matrix transpose, λ is a hyperparameter, and λ=0.2; S2.3 uses multiple heads to enhance the cross-attention difference calculation in step S2.2, learns relevant judicial knowledge from different dimensions, and obtains multi-head cross-attention difference results in, Indicates the Q and K parameters for calculating the attention value of the i-th head, W O ∈R d×d is a learnable weight matrix, Indicates the attention value of the i-th head, DiffAttn i represents the cross-attention difference result of the i-th head, MultiHeadAttn(·) represents the scaled dot product multi-head attention, and LN(·) represents the linear normalization processing; S2.4 In order to reduce the loss of case fact information due to network depth during the judicial knowledge fusion process, a residual connection is performed after the multi-head cross-attention difference calculation, which is calculated as follows: in, is the case fact feature representation enhanced by judicial knowledge, Res(·) represents the residual connection processing; The step S3 specifically includes: S3.1 Representing case fact features enhanced by judicial knowledge Input into an LSTM long short-term memory network consisting of three time steps, respectively for the legal clause prediction subtask in the legal judgment l , accusation prediction subtask c and sentence prediction subtask t Make predictions and define the following masked task dependencies for these three subtasks: task l ≥task c 、task l ≥task t and task c ≥task t , where the symbol "≥" represents a masked task dependency, which means that during the training phase, the features of the predecessor task are transferred to the subsequent task only when the prediction of the predecessor task is true, otherwise they are not transferred; if the features of the predecessor task are transferred successfully, the subsequent task depends on the features of the predecessor task for prediction; S3.2 Use the first time step in the LSTM network to predict the law task l Perform calculations and predictions: First, calculate the legal clause prediction task l The decoded hidden state of : Among them, LSTMCell1(·) represents the first time step in the LSTM network, Represents the initial hidden state and initial memory unit of LSTMCell1, H l ∈R d Represents the legal clause prediction task obtained after processing by LSTMCell1 l The decoded hidden state, Z l Represents the value of the memory cell after processing by LSTMCell1; Then, a fully connected layer that performs the Softmax function is used to calculate the law prediction task task l The prediction results are: Among them, W l ∈R m×d 、b l ∈R m They are tasks l The trainable weight matrices and biases in the task, is the probability distribution on the legal article labels, is the predicted label of the legal clause prediction task, Express request The legal label corresponding to the element with the highest probability, i1 represents The element subscript in , express The i1th element in ; Finally, according to Whether the prediction is correct determines whether the task l Task information is updated for subsequent subtasks: Among them, when When, task l If the task prediction result is correct, then Mask l The value is True, otherwise it is False, the l t To predict the sample in task l The actual label in the task, Mask l For task l The mask for task information to be passed back; S3.3 Use the second time step in the LSTM network to predict the accusation task c Perform calculations and predictions: First, Mask l Applied to the following calculations to achieve the task l Filtering of incorrect task dependency information in tasks: Among them, W l，c ∈R d×d 、b l,c ∈R d Through task l Task information update task c The initial hidden state of the task and the weight matrix and bias of the memory unit, Respectively represent tasks c The initial values of hidden states and memory units in the task; when Mask l =True, Why use formula (30) to calculate, otherwise Then, use the following formula to calculate the charge prediction task task c The decoded hidden state of : Among them, LSTMCell2(·) represents the second time step in the LSTM network, H c ∈R d Indicates the charge prediction task obtained after processing by LSTMCell2 c The decoded hidden state, Z c Represents the value of the memory unit after LSTMCell2 processing; Then, a fully connected layer that performs a Softmax function is used to calculate the charge prediction task. c The prediction results are: Among them, W c ∈R k×d 、b l ∈R k It's a task c The trainable weight matrices and biases in the task, is the probability distribution over the accusation labels, is the predicted label of the accusation prediction task, i2 represents The element subscript in , express The i2th element in ; Finally, according to Whether the prediction is correct determines whether the task c Task information is updated for subsequent subtasks: Among them, when When, task c If the prediction result of the task is correct, then Mask c The value is True, otherwise it is False, the c t Indicates that the predicted sample is in task c The actual label in the task, Mask c For task c The mask for task information to be passed back; S3.4 Use the third time step in the LSTM network to predict the sentence task t Perform calculations and predictions: First, Mask l and Mask c Applied to the following calculations to achieve the task l Legal clause prediction tasks and tasks c Filtering of incorrect task-dependent information in the accusation prediction task: Among them, W l，t ∈R d×d 、b l，t ∈R d Through task l Task information update task t The initial hidden state and memory unit weight matrix and bias for the sentence prediction task, W c，t ∈R d×d 、b c，t ∈R d Through task c Task information update task t The initial hidden state and memory unit weight matrix and bias for the sentence prediction task, task t The initial values of hidden states and memory units in the sentence prediction task; only when Mask l =True, Mask c = True, the corresponding Information is involved in the calculation of formula (35). If Mask l = False and Mask c = False, then Then, use the following formula to calculate task t Decoded hidden state for the prison sentence prediction task: Among them, LSTMCell3(·) represents the third time step in the LSTM network, H t ∈R d Indicates the task obtained after LSTMCell3 processing t Decoded hidden state for sentence prediction task, Z t Indicates the value of the memory cell after LSTMCell3 processing; Finally, a fully connected layer that performs a Softmax function is used to calculate the sentence prediction task t The prediction results are: Among them, W t ∈R j×d 、b t ∈R j It's a task t Trainable weight matrices and biases for the sentence prediction task, is the probability distribution over sentence labels, j is the number of sentence labels in the corpus, is the predicted label of the sentence prediction task, i3 represents The element subscript in , express The i3th element in .

Images