Legal text named entity recognition method based on small sample and fusion of knowledge tips

Abstract

The application relates to a legal text small sample named entity recognition method fusing knowledge prompts and belongs to the field of natural language processing and machine learning. The application firstly defines the mapping relationship of the suggestive template, the label set and the natural word set of the judicial documents, obtains 2760 pieces of ruling and judgment document from the Chinese judicial document network platform; then encodes the embedding vectors of the case statement and the judgment result sentence by adopting a Bert model, obtains the feature representation of the sentence through attention weighting, and generates the category vector of the word in combination with the template guided generative pre-training model GPT; finally, the sentence vector is converted into a span vector by using a full connection layer, a semantic label is generated, and the loss function of the semantic label and the label vector is minimized. The application effectively alleviates the overfitting and inaccurate classification problems caused by the small amount of labeled samples and the different entity type distribution in the legal field, and improves the accuracy of the legal entity recognition and the migration ability of the model.

Description

TECHNICAL FIELD

[0001] The application relates to a legal text small sample named entity recognition method fusing knowledge prompts and belongs to the field of natural language processing and machine learning. BACKGROUND

[0002] Named entity recognition is a basic task in natural language processing, aiming to obtain entities from text and classify them by labels, and it is widely used as the basis for downstream tasks such as information extraction and machine translation. At present, named entity recognition technology has achieved good results in limited fields and entity types, but these technologies cannot be well transferred to other specific fields.

[0003] With the increasing demand for specific natural language processing tasks, applying named entity recognition methods in the legal field to obtain entities with field characteristics such as plaintiff and defendant names, case attributes, law basis, responsibility bearing, and involved property can assist judges in decision-making and realize a smart courtroom. Therefore, researching named entity recognition tasks in the legal field is of application value.

[0004] Current named entity recognition methods based on legal texts include three categories: methods based on domain similar text representation, which rely on similar text representation between source domain and legal domain and do not fully consider the particularity of legal domain entities; methods based on domain dictionary, which consider model transferability but have high cost of manually setting domain dictionary; and methods based on neural networks CNN, RNN, LSTM, and Bert pre-training model, such as the Bert-BiLSTM-CRF mainstream model applied to legal domain entity recognition tasks, but this method lacks high-quality legal domain samples, has low accuracy in recognizing legal entities, and has high cost in training models, so there are still challenges in the research of legal small sample named entity recognition.

[0005] In summary, the existing named entity recognition methods for legal texts mainly have the following problems: (1) the method based on text similarity representation does not update network parameters, cannot improve cross-domain instance representation, reduces the domain adaptation ability, and does not fully consider the deep semantic information of entity labels; (2) the method based on domain dictionary uses enumeration to generate candidate entities, which has serious efficiency problems and does not fully exploit the relationship between legal entity labels; (3) there are few public legal text datasets, and the model training relies on annotated corpus, which can easily cause overfitting problem when training on small samples. SUMMARY

[0006] The purpose of the application is to fuse the prompt learning method, construct the template for the small sample in the legal field, and optimize the label classification effect by means of the template for fine-tuning the pre-trained model. In view of the great difference between the entity boundary and the type distribution in the legal field and the entity in the general field, the application introduces the professional knowledge in the legal field to construct the entity and non-entity knowledge template, uses the knowledge template to guide the pre-trained model, performs prompt and fine-tuning, optimizes the classifier, maps the label to a wider semantic space, avoids the retraining of the parameters of the deep learning model, reduces the demand for labeled data, fully utilizes the semantic information of the entity and label in the template, and improves the small sample learning potential of the language model.

[0007] The design principle of the application is as follows: first, obtaining original data of judgment documents from the Chinese judgment document network, preprocessing the judgment documents; second, defining positive sample templates and negative sample templates of the judgment documents, defining a label set and a natural word set; and finally, constructing a judgment document named entity recognition model, inputting the sentence level judgment document into a Bert pre-training model, outputting a sentence feature vector, connecting the judgment document vector and the template vector, using a GPT generative pre-training model to decode, predicting the mask in the template to generate a task, combining the label feature vector to minimize the loss function, and optimizing the entity classification effect.

[0008] The technical scheme of the application is realized through the following steps:

[0009] Step 1, obtaining original judgment document data from the Chinese judgment document network, predefining five types of named entities of the judgment documents, which are the plaintiff and the defendant (Person), the case attribute (Case), the regulations (Regulations), the responsibility (Responsibility) and the involved property (Compensate).

[0010] Step 2, based on a large amount of entity and non-entity knowledge in the judgment documents, constructing a prompt template, a label set and a natural word set mapping relationship.

[0011] Step 2.1, constructing positive and negative example templates of entity knowledge by artificial means to guide the pre-training model for fine-tuning.

[0012] Step 2.2, predefining the entity category label set L={l1,l2,…,l |L|} of the model space and the natural word set Y={y1,y2,…,y |L|} of the semantic space.

[0013] Step 3, constructing a judgment document named entity recognition model to identify five types of legal entities existing in the judgment and decision documents.

[0014] Step 3.1, each word in the judgment document is expressed as a token embedding vector and an embedding vector recording the time attribute of each word, and the two vectors are combined and input into the Bert pre-training model, and the representation W of the i-th sentence is output i .

[0015] Step 3.2, use the GPT generative pre-training model as the latent variable decoder, combine the sentence vector output in step 3.1, obtain the feature representation of the current element label based on the semantic information of the current element and the forward element in the template, and enhance the feature vector representation based on the prompt template.

[0016] Step 3.3, according to the feature representation vector of the label in the sentence, the sentence vector is converted into a span vector by using a full connection layer, the non-normalized probability of the entity feature belonging to five categories of labels is calculated, the loss function of the model is defined, the class label with higher confidence is generated, and the label classification effect is optimized.

[0017] Beneficial effects

[0018] Compared with the method of Bert-BiLSTM-CRF, the present application guides the fine-tuning of the pre-training language model based on the templates and labels in the legal field, reduces the demand for labeled data and the cost of model training, optimizes the label classification result, and fully considers the class words with relatively close distance in the semantic space. The constructed template and label are suitable for small sample tasks in the legal field, and the five types of entities in the judgment document, namely the plaintiff, the defendant, the case attribute, the law basis, the responsibility assumption and the involved property, are realized, and the entity recognition accuracy, model migration and field adaptation ability in a specific field are improved. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 The principle diagram of the legal text small sample named entity recognition method of the present application fusing knowledge prompts. DETAILED DESCRIPTION

[0020] In order to better illustrate the purpose and advantages of the present application, the embodiments of the method of the present application will be further described in detail in combination with examples.

[0021] The experiment was carried out on a computer and a server. The specific configuration of the computer is: Inter i7-6700, CPU 2.40GHz, memory 16G, operating system is windows 10, 64 bit; the specific configuration of the server is: E7-4820v4, RAM 256G, operating system is Linux Ubuntu 64 bit.

[0022] The experimental data were obtained from the China Judgments Online database, including two types of datasets: rulings and judgments. Based on the named entity recognition experimental data of legal documents, the total number of ruling documents was 2060, and the total number of judgment documents was 700. The training sets for these two types of documents were 2000 and 600 respectively. Ten documents from each type were randomly selected for model validation and testing.

[0023] The experiment used the F1 score to evaluate the results of entity label classification in the judgment documents. The calculation method is shown in formula (1).

[0024]

[0025] The calculation methods for precision and recall are shown in formulas (2) and (3).

[0026]

[0027]

[0028] In this model, TP is the number of correct entities correctly predicted, FN is the number of non-entities predicted as negative, and FP is the number of non-entities correctly predicted as entities. The F1 score is the harmonic mean of precision and recall, and is used as an evaluation metric for the performance of named entity recognition in court documents. The higher the F1 score, the better the model's performance.

[0029] The specific procedure for this experiment is as follows:

[0030] Step 1: Obtain original judgment data from the China Judgments Online website, predefine named entities for the judgments, and process both rulings and judgments by removing stop words. Based on a publicly available stop word database and TF-IDF word frequency statistics, remove meaningless words in the legal field, such as "application," "ruling," and "in accordance with the law," to obtain the processed judgment data. Analysis of the statistical results of the original judgment data shows that the entities involved in civil case rulings and judgments include five categories: the names of the plaintiff and defendant, the nature of the case, the legal provisions applied, liability, and the property involved.

[0031] Step 2: Based on the entity and non-entity knowledge in a large number of judgment documents, construct a positive example template T. + and negative example template T - Mapping relationship between tag set and natural word set.

[0032] Step 2.1, based on a large amount of court judgment data k represents the k-th sentence, and N represents the number of words in the sentence. This invention uses manually constructed entity knowledge templates Template = [T + ,T- ], which improves the effect of judgment document entity recognition. Among them, T + is set to "[X1], [X2] is a [MASK] entity." T - is set to "[X1], [MASK] is not an entity." [X1] represents the sentence-level judgment document [X2] is the entity feature vector obtained by the pre-training model, and [MASK] is the entity category label to be predicted and generated. For example, the template instance is set to "[X1], infringement dispute series is a [MASK] entity", where the infringement dispute series is [X2].

[0033] According to the characteristics of the plaintiff and defendant names, case attributes, law articles, responsibility assumption and involved property of the judgment documents, the entity knowledge and non-entity knowledge are used to define the template instances, as shown in Table 1.

[0034] Table 1. Domain knowledge template instance

[0035]

[0036] In the template instance table, [MASK] of the three positive example templates is filled with "case attribute", "law article" and "responsibility assumption" respectively.

[0037] Step 2.2, predefine the entity category label set L = {l1, l2, …, l |L|} of the model space and the natural word set Y = {y1, y2, …, y |L|} of the semantic space, Y = {Person, Case, Regulations, Responsibility, Compensate}, the type of the person name entity matches the external corpus. Among them, |L| represents the number of entity categories, and the label set and the natural word set correspond to each other.

[0038] Step 3, build a judgment document named entity recognition model composed of a Bert pre-training model and a GPT generative pre-training model. The sentence-level judgment document in step 1 is input into the model, and the word vector representation of each sentence is output. Through the template fine-tuning model parameters constructed in step 2.1, the sentence feature vector fused with template information is output.

[0039] Step 3.1, obtain the token embedding vector U and the position embedding vector P of the sentence-level judgment document, n is the number of words in the sentence, let Z = U + P, Z is the combined embedding vector, input Z into the Bert pre-training model, and the calculation process of each head in the self-attention layer is shown in formulas (4), (5), (6).

[0040]

[0041]

[0042]

[0043] where, denote the query weight matrix, key weight matrix, value weight matrix, query bias vector, key bias vector, value bias vector of the i-th head, respectively, d k = 64 denotes the dimension of Q, K, V.

[0044] The feature information of the bottom layer is fused, that is, the attention weight matrix A is calculated by combining the Q, K, V matrices, and the calculation process is shown in formula (7).

[0045]

[0046] Each element a ij in the matrix A represents the weight parameter between q i and k j . For the i-th word z i , the hidden layer state and attention weight calculated respectively in the forward and backward directions are combined, that is, the new hidden layer state h n is obtained, and the representation of the i-th sentence is output: W i = {w i,1 ,w i,2 ,w i,3 ,…,w i,n}.

[0047] Step 3.2, after training the Bert model, the GPT generative pre-training model is used as the hidden variable decoder. Since there is [MASK] in the template, the probability of the i-th word appearing is predicted based on the current element and the element before the mask, combined with the hidden variable h n and the token embedding vector in step 3.1, and the calculation process is shown in formulas (8) and (9).

[0048] h0=UW te +W pe (8)

[0049]

[0050] W te denotes the token embedding matrix, and W pe denotes the position embedding matrix. The conditional probability of the word w i is calculated, the category information at [MASK] is predicted according to the probability value, and the template prompt-based fine-tuning operation is realized.

[0051] The probability of the label of the t-th token in the k-th sentence belonging to the L class (L = 1, 2, 3, 4, 5) is represented as [f t,1 (X k ; h n ), f t,2 (X k ; h n ), f t,3 (X k ; h n ), f t,4 (X k ; h n ), f t,5 (X k ; h n )]. Among them, f t,L (X k ; h n ) represents the probability of the t-th token belonging to the L class, and the semantic label generation process is shown in formulas (10) and (11).

[0052]

[0053]

[0054] wherein k represents the number of sentences in the document, n represents the number of words in the sentence, M k represents the semantic label space of the k-th sentence, m k,n represents the label subspace to which the n-th token in the k-th sentence belongs, which is composed of different token class labels, The non-normalized probability of the token belonging to the l-th class is calculated, and the class label with high confidence is generated.

[0055] Step 3.3, by minimizing the loss function, the class label with high confidence is predicted. The loss function is shown in formula (12).

[0056]

[0057] wherein l represents the number of entity classes, if the true label of the n-th word in the k-th sentence is l, then m k,n,l = 1, otherwise, m k,n,l = 0; f n,l (X k ; h [MASK] ) represents the probability of the n-th word belonging to the l-th entity predicted by the model, and the Adam algorithm is used to optimize the entity label classification result.

[0058] The application uses a knowledge template, a label word to guide a pre-training model, fine-tunes model parameters, combines a template prompt to output an entity category label, and maps the label to a semantic vector space. Randomly extracting 20 documents for testing, the experimental results show that all original defendants, case attributes, law articles, responsibility bearing and involved properties can be correctly identified. Compared with other entity identification methods in general and legal fields, the model has higher accuracy in identifying entities, greatly reduces the model training cost, and improves the recall rate and the performance of the model.

[0059] The above specific description further details the purpose, technical solutions and beneficial effects of the application. It should be understood that the above description is only a specific embodiment of the application and is not used to limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the application should be included in the protection scope of the application.

Claims

1. A named entity recognition method for small samples of legal texts that integrates knowledge prompts, characterized in that... The method includes the following steps: Step 1: Obtain original rulings and judgments from the China Judgments Online website. Construct five named entities for the judgments: plaintiff and defendant names, case attributes, legal provisions, liability, and involved property. Remove stop words to obtain the processed civil case rulings and judgments. Step 2: Construct the mapping relationship between entity and non-entity knowledge templates, tags, and natural words in the judgment documents; Using templates from the legal field The BERT pre-trained model is guided by a set of labels in the semantic space, based on the characteristics of five categories of entities in the judgment documents: the names of the plaintiff and defendant, the nature of the case, the legal provisions, the liability, and the property involved. Let "[X1], [X2] is a [MASK] class entity". Let "[X1], [MASK] is not an entity", where [X1] represents a sentence-level legal document, [X2] is the entity feature vector obtained from the pre-trained model, and [MASK] is the entity category label to be predicted. Step 3: Construct a named entity recognition model for court judgments. The model consists of a BERT pre-trained model, a GPT generative pre-trained model, and fully connected layers. Sentence-level court judgment sequences are input into the model. BERT encodes the sentences, and the GPT model decodes them. The pre-trained model is fine-tuned using prompt templates to predict entity category labels at the generated mask. Label mapping is performed for each entity category, and a loss function is constructed to optimize the label classification results. Specifically, this includes incorporating the latent variables from the last layer of the BERT pre-trained model. and legal templates Using the GPT generative pre-trained model, the formula is... and Predict the entity type at the mask, where U represents the token embedding vector. This represents a token embedding matrix. Represent the position embedding matrix; then use the formula Calculate the nonnormalized probability of a token belonging to class l. Let K represent the set of entity tags, K represent the number of sentences in the document, N represent the number of characters in each sentence, and k and n represent the k-th sentence and the n-th character, respectively. Let represent the probability that the t-th token belongs to the L-th class; the formula for generating the semantic tag space is: The semantic label space representing the k-th sentence; , Represents the category in the entity label set L. This indicates that the t-th token is the t-th token. The probability of a class Let represent the label subspace to which the nth token in the kth sentence belongs, composed of category labels for different tokens; finally, the label classification result is optimized using a loss function. Let l represent the loss function for label classification, where l represents the number of entity categories. If the true label of the nth word in the kth sentence is l, then... ,otherwise, ; This represents the probability that the nth word, as predicted by the model, belongs to the lth type of entity.

Images