Named entity recognition method and system in the judicial field based on BERT model

Abstract

The present invention discloses a method and system for named entity recognition in the judicial field based on the BERT model. Traffic accident judgments in the judicial field are selected as data sets, and the data sets are processed by a remote supervision method to realize the labeling of the data sets; the partially labeled data sets are input into the pre-trained model BERT‑CNN for fine-tuning, thereby transferring the semantic knowledge in BERT and improving the prediction quality of remote supervision. At the same time, early stopping measures are taken during the training process to prevent the model from overfitting to incomplete labeled data. Finally, the BERT‑CNN model is used to predict a set of soft labels for all data; the newly generated data set with soft labels replaces the data set generated by the remote supervision method, and a teacher‑student framework is designed to realize named entity recognition and further improve the recall rate.

Description

Technical Field

[0001] The present invention relates to a method and system for identifying named entities in the judicial field based on a BERT model, and belongs to the technical field of text data or knowledge extraction. Background Art

[0002] With the advent of the big data era, information is growing explosively. How to obtain effective information from massive amounts of information data is the research content of knowledge extraction technology. As one of the important subtasks of knowledge extraction, named entity recognition technology is receiving more and more attention. As a typical knowledge-intensive industry, the judicial field requires the storage of entity information such as laws, regulations, cases, and parties in judicial work. Traditional methods rely on manual extraction and integration of information, which is time-consuming and laborious, and cannot meet the needs of massive information extraction. Therefore, designing models to automatically identify named entities in the judicial field has become a hot issue in the current judicial industry.

[0003] Currently, named entity methods are mainly divided into three categories:

[0004] The first category is the rule-based method, which first constructs a large number of entity extraction rules, which are generally manually constructed by experts with certain domain knowledge. Then, the rules are matched with text strings to identify named entities. The second category is the statistical method, which uses fully annotated or partially annotated corpus for model training, mainly involving three aspects: training corpus annotation, feature definition, and model training. The third category is the method based on deep learning, which directly uses the vector of words in the text as input and realizes end-to-end named entity recognition through neural networks, no longer relying on manually defined features.

[0005] However, the training process is mostly oriented towards structured data, which requires a large-scale well-labeled data set. Faced with the massive text corpus in the judicial field, the method of manually labeling is obviously not feasible. In order to solve the problem of label scarcity, existing related research usually uses remote supervision to solve it. This method obtains data labels by aligning the text with the existing knowledge base. However, there is a problem of noisy labels, which leads to low quality of labels generated by the matching process. At the same time, most existing models use traditional neural network models for encoding, which only considers the dependency relationship in a single direction of language during encoding, resulting in insufficient semantic feature extraction. Therefore, how to improve the ability of semantic feature extraction and eliminate noise labels is an urgent problem to be solved in named entity recognition methods in the judicial field. Summary of the invention

[0006] Purpose of the invention: In view of the problems and shortcomings in the prior art, the present invention provides a method and system for named entity recognition in the judicial field based on the BERT model, which can improve the ability of semantic feature extraction and address the problem of noise labeling caused by remote supervision methods.

[0007] Technical solution: A method for named entity recognition in the judicial field based on the BERT model, comprising the following steps:

[0008] Step 1) Select traffic accident judgments in the judicial field as the data set, use the remote supervision method to process the data set, and realize the labeling of the data set.

[0009] Step 2) Input part of the labeled dataset into the pre-trained model BERT-CNN for fine-tuning, thereby transferring the semantic knowledge in BERT and improving the prediction quality of remote supervision. At the same time, early stopping measures are taken during the fine-tuning process to prevent the model from overfitting to incomplete labeled data. Finally, the BERT-CNN model is used to predict a set of soft labels for all data.

[0010] Step 3) replaces the dataset generated by remote supervision in step 1) with the newly generated dataset with soft labels. In order to improve the prediction accuracy, a teacher-student framework is designed to further improve the recall rate by continuous training iteration. The student model is initialized by the model of the soft label dataset trained in step 2), and the teacher model is updated by the student model in the previous iteration.

[0011] Step 4) The named entity recognition method model based on the BERT model obtained after training in step 3) is applied to the unstructured case text in the judicial field to automatically identify relevant entities.

[0012] Among them, the remote supervision method used in step 1) to process the judicial field dataset includes the following process:

[0013] 1-1) Crawl a large amount of unstructured judicial text data from the Internet.

[0014] 1-2) Use the existing word segmentation tools to segment the text data crawled in 1-1) into sentences and words, and then use the named entity recognition tool to perform entity annotation on the segmented text data in units of sentences.

[0015] 1-3) Using an external knowledge base as a supervision source, the entities marked in step 1-2) are matched with the concepts in the knowledge base (such as Wikipedia, YAGO, etc.), and relevant dictionary information is collected from multiple online resources to label the entities in the sentence.

[0016] Among them, the fine-tuning of the pre-trained model BERT-CNN in step 2) means: encoding the data set annotated in step 1) and using the pre-trained model BERT-CNN to complete the semantic feature extraction of the sentence. Specifically, it includes the following steps:

[0017] 2-1) Input the labeled dataset in step 1) into the model and convert it into an input vector form consisting of three word embeddings added together, which is the input form required by the BERT model (BERT-CNN combines BERT and CNN in sequence, and the input corpus first passes through the BERT model). The input vector form consisting of three word embeddings added together, where: the first is to use the WordPiece model to convert the input sentence into a word vector form; the second is to use Segment Embedding to determine whether there is a contextual relationship between two sentences; the third is to obtain the word position information through Position Embedding.

[0018] 2-2) Since the length of each sentence input during training is inconsistent, when the length of the input sentence is less than the pre-defined maximum length (max_length), it is necessary to fill it with 0. The Masked Language Model mechanism is used during training. The masked language model can be understood as a cloze test, which randomly masks part of the words in the sentence and uses the context of the instance to make predictions. It ensures that [mask] is used in the masked part 80% of the time, a random word is taken to replace the masked word 10% of the time, and it remains unchanged 10% of the time. Finally, the input vector is input into the BERT model to obtain the low-dimensional vector representation corresponding to each sentence.

[0019] 2-3) The low-dimensional vector processed by the BERT model is connected to the CNN network model to obtain the final semantic feature vector. Among them, the CNN network model includes three convolution modules and activation functions. The feature vector obtained in 2-2) is reduced in dimension through the pooling operation of the CNN network model, and then passed through a linear layer and a dropout layer to alleviate the overfitting of the network. During the training process of the CNN network model, an early stopping mechanism is also adopted. Specifically, when the performance of the model on the validation set begins to decline, the training is stopped immediately to prevent the model from overfitting the data and improve the generalization ability of the data. Among them, the activation function is expressed as:

[0020]

[0021] v represents the feature vector in the network structure.

[0022] 2-4) Use the BERT-CNN model fine-tuned in step 2-3) to train the entire dataset, use the softmax classifier to classify the processed semantic feature vectors, and predict a set of soft labels for all data.

[0023] Step 3) builds the teacher-student framework including:

[0024] 3-1) The model fine-tuned by the dataset is used as the initial teacher model, expressed as The model directly trained by the pre-trained model BERT-CNN without fine-tuning is initialized as the student model, which is expressed as

[0025] 3-2) Named entity recognition is the process of classifying named entities in a text into predetermined entity types. Formally, given a sentence with N tokens (labels), the entity label is related to the entity type.

[0026] The teacher model proposed in 3-1) uses the formula Generate pseudo labels, perform iterative updates, and optimize them, where f n，c represents the probability that the nth token belongs to the cth class, X m Represented as the mth sentence; the student model uses the formula To fit these labels;

[0027] 3-3) Using ADAM algorithm to optimize the formula And according to the formula The student model and the teacher model are iteratively updated. After multiple iterations, the entity category classified by the student model is used as the final entity category.

[0028] The judicial named entity recognition system based on the BERT model includes the following modules:

[0029] Dataset annotation module: Traffic accident judgments in the judicial field are selected as the data set, and the data set is processed using remote supervision methods to achieve data set annotation.

[0030] Pre-trained model BERT-CNN fine-tuning module: Part of the labeled data set is input into the pre-trained model BERT-CNN for fine-tuning. Early stopping measures are taken during the fine-tuning process, and the BERT-CNN model is used to predict a set of soft labels for all data.

[0031] Teacher-student model training module: replace the dataset generated by the remote supervision method with the newly generated dataset with soft labels; design a teacher-student framework to further improve the recall rate through continuous training iterations; the student model is initialized by the model of the soft-label dataset, and the teacher model is updated by the student model in the previous iteration.

[0032] Entity recognition module: The named entity recognition method model based on the BERT model obtained after training in the teacher-student model training module is applied to unstructured case texts in the judicial field to automatically identify relevant entities.

[0033] The implementation process of the system is the same as the above method.

[0034] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for named entity recognition in the judicial field based on the BERT model as described above is implemented.

[0035] A computer-readable storage medium stores a computer program for executing the method for named entity recognition in the judicial field based on the BERT model as described above.

[0036] Compared with the prior art, the advantages of the present invention are:

[0037] 1) The named entity recognition method described is different from the traditional named entity recognition method and uses a remote supervision method to annotate the data set. This method is mainly used for named entity recognition of unstructured data in the judicial field and has certain practicality.

[0038] 2) The named entity recognition method introduces the BERT-CNN model for semantic feature extraction of sentences. Based on the advantages of the BERT model and the CNN model, the problem of insufficient long-distance feature extraction capability and insufficient extraction of ordinary networks is solved, and high-quality feature extraction of lengthy sentences is achieved;

[0039] 3) The named entity recognition method described above reduces noise by constructing a teacher-student framework to address the possible noise problem in the data set. This method iterates the teacher and student models respectively, continuously optimizes the generated soft labels, and ultimately achieves the effect of noise reduction. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1The overall framework diagram of the method for named entity recognition in the judicial field based on the BERT model provided in the embodiment of the present invention;

[0041] Figure 2 A flow chart of a remote supervision method provided by an embodiment of the present invention;

[0042] Figure 3 A schematic diagram of a BERT pre-trained language model provided in an embodiment of the present invention;

[0043] Figure 4 A schematic diagram of a CNN model provided by an embodiment of the present invention;

[0044] Figure 5 A schematic diagram of a teacher-student optimization framework provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0046] Figure 1 The overall framework diagram of the judicial field named entity recognition method based on the BERT model provided by the present invention, and its working process is as follows:

[0047] Step 1) Obtain a large amount of unstructured text corpus in the judicial field and preprocess it. Segment the content in it, construct a special data set for judicial cases, and use remote supervision methods to annotate entities.

[0048] The specific process is:

[0049] 1-1) A large amount of unstructured text data is crawled from the China Judgments Online website, including first-instance judgments of criminal, civil, administrative, compensation, execution and other related cases. The text data of this embodiment selects the first-instance judgments of traffic accident cases as the experimental data set.

[0050] 1-2) Segment the text in 1-1) into independent sentences, and segment the text in units of sentences with the help of the existing word segmentation tool jieba. Then use the existing named entity recognition tool to annotate the text data after word segmentation in units of sentences. The entity categories include natural persons, non-natural persons, motor vehicles, non-motor vehicles, insurance, liability determination, personal injury types, time, property damage types, and illegal acts, among which insurance is divided into commercial insurance and compulsory traffic insurance; liability determination is divided into full liability, no liability, primary liability, and secondary liability; personal injury types are divided into general personal injury, disability, and death; property damage types are divided into damage and destruction; illegal acts include driving without a license, escaping, overloading, speeding, driving under the influence of alcohol, and driving under the influence of alcohol, etc.

[0051] Table 1 below lists the entity types predefined in this embodiment:

[0052] Table 1

[0053] serial number Entity Type Name serial number Entity Type Name 1 Natural Person 6 Responsibility determination 2 Non-natural person 7 Types of personal injury 3 Motor Vehicles 8 Item damage type 4 Non-motorized vehicles 9 time 5 Insurance 10 Illegal behavior

[0054] For example, the tool is used to perform entity recognition on the sentence "In February 2019, the defendant Zhang San was driving a motorcycle and collided with the pedestrian Li Si", where "defendant Zhang San", "pedestrian Li Si", "motorcycle", and "February 2019" are all extracted entities.

[0055] 1-3) Using the external knowledge base as a supervision source, the entities marked in step 1-2) are matched with the concepts in the knowledge base (such as Wikipedia, YAGO, etc.), and relevant dictionary information is collected from multiple online resources to label the entities in the sentence. The specific working process is as follows Figure 2 As shown in the figure: first, select the entities in the unlabeled judicial field dataset; secondly, enter the entity name into Wikipedia and other online network resources to query whether there is relevant information about the entity name; then determine whether there is information about the entity name. If so, label it for annotation; if not, terminate the query; finally, output the labeled entity information after the query to complete the entire remote supervision labeling process.

[0056] For example, the entity input obtained by the named entity tool in 1-2) is matched through Wikipedia and other relevant online resources to return the specific label information of the entity, such as ("defendant Zhang San", wd:Q10000), ("pedestrian Li Si", wd:Q10001), ("motorcycle", wd:Q10002);

[0057] Step 2) encodes the labeled dataset in 1-3), and uses part of the data to fine-tune the model, and applies the adjustment to solve the named entity recognition problem. Figure 3Schematic diagram of the BERT model of the method of the present invention.

[0058] The internal structure of the model mainly consists of four parts: Multi-Head Attention, Add&Norm, Feed Forward, and Position Encoding. Multi-Head Attention is the most important module of the model, and the calculation formula is:

[0059] Among them, Q, K and V are the input vector matrices, d k is the input vector dimension. It combines the characteristics of Multi-Head and attention, so that each word in the sentence contains information about other words, and can be processed in parallel, which allows the model to focus on information from different representation subspaces at different positions, thereby extracting more sentence features; Add&Norm (regularization operation) uses residual network and normalization operation to speed up model convergence and alleviate overfitting; FeedForward (feedforward neural network layer) performs full connection operation on the feature vector extracted by the multi-head self-attention layer through two fully connected layers; Position Encoding (position encoding) ensures that the parallel execution network learns the position information between words.

[0060] Figure 4 The figure is a schematic diagram of the CNN model of the method of the present invention. Through three convolution modules of different sizes and the GeLU activation function, the semantic features of the sentence are further extracted. By combining the pre-trained language model with the CNN model, the generalization ability of the model can be further enhanced, and the semantic information in the sentence can be fully extracted.

[0061] At the same time, the early stopping mechanism is used in the fine-tuning process. Specifically, when the performance of the model on the validation set begins to decline, the training is stopped immediately to prevent the model from overfitting the data and improve the generalization ability of the data.

[0062] For example, some labeled data sets (such as "defendant Wang Wu" and "Toyota Motors" etc.) are input into the BERT-CNN model for pre-training, and then the BERT model and CNN model are used, and the accuracy changes of the models are constantly observed during the verification process. When the performance of the model on the verification set decreases, the training is stopped immediately, and the model obtained at this time is the fine-tuned model.

[0063] Step 3) Use the fine-tuned BERT-CNN model obtained in step 2) to train the entire dataset and predict a set of soft labels for all data, including:

[0064] 3-1) The teacher-student framework module uses the fine-tuned BERT-CNN model as the initial teacher model and initializes the unfine-tuned BERT-CNN model as the student model. The entire framework diagram is shown below. Figure 5 As shown. The teacher model uses the formula Generate pseudo labels and perform iterative updates, where And it can be optimized; the student model uses the formula To fit these labels;

[0065] 3-2) Use the ADAM algorithm to optimize the formula and iteratively update the student model and teacher model. After multiple iterations, the entity category classified by the student model is used as the final entity category.

[0066] For example, "defendant Zhang San" and "pedestrian Li Si" are classified as natural persons, "motorcycle" is classified as a motor vehicle, and "February 2019" is classified as a time.

[0067] Step 4) The named entity recognition model trained in step 3-2) is used for named entity recognition of unstructured case texts in the judicial field. An unlabeled sentence is input and the model is used to finally obtain the predicted entity types of all entities in the sentence.

[0068] According to the above embodiments, the present invention implements a method for named entity recognition in the judicial field based on the BERT model. According to the experimental requirements, the judicial field scenario is set, and the judicial field traffic accident judgment is selected as the data set. The present invention uses a remote supervision method to annotate the data set, and uses the model BERT-CNN and the teacher-student framework to realize named entity recognition. The data set is passed through a named entity recognition tool, and the entities identified therein are labeled using a remote supervision method. The pre-trained model BERT-CNN uses the annotated data set to perform fine-tuning and predict a set of soft labels; it uses the teacher-student model framework for iterative updates to solve the noise problem caused by the remote supervision method.

[0069] The judicial named entity recognition system based on the BERT model includes the following modules:

[0070] Dataset annotation module: Traffic accident judgments in the judicial field are selected as the data set, and the data set is processed using remote supervision methods to achieve data set annotation.

[0071] Pre-trained model BERT-CNN fine-tuning module: Part of the labeled data set is input into the pre-trained model BERT-CNN for fine-tuning. Early stopping measures are taken during the fine-tuning process, and the BERT-CNN model is used to predict a set of soft labels for all data.

[0072] Teacher-student model training module: replace the dataset generated by the remote supervision method with the newly generated dataset with soft labels; design a teacher-student framework to further improve the recall rate through continuous training iterations; the student model is initialized by the model of the soft-label dataset, and the teacher model is updated by the student model in the previous iteration.

[0073] Entity recognition module: The named entity recognition method model based on the BERT model obtained after training in the teacher-student model training module is applied to unstructured case texts in the judicial field to automatically identify relevant entities.

[0074] Obviously, those skilled in the art should understand that the various steps of the judicial field named entity recognition method based on the BERT model or the various modules of the judicial field named entity recognition system based on the BERT model of the above-mentioned embodiment of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented with program codes executable by the computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules respectively, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

Claims

1. A method for named entity recognition in the judicial field based on the BERT model, characterized in that: The steps include: Step 1) Select traffic accident judgments in the judicial field as a data set, use remote supervision methods to process the data set, and realize the labeling of the data set; Step 2) Part of the labeled data set is input into the pre-trained model BERT-CNN for fine-tuning. Early stopping measures are taken during the fine-tuning process, and a set of soft labels are predicted for all data using the BERT-CNN model; Step 3) replaces the dataset generated by the remote supervision method in step 1) with the newly generated dataset with soft labels; designs a teacher-student framework to further improve the recall rate by continuous training iteration; wherein the student model is initialized by the model of the soft label dataset trained in step 2), and the teacher model is updated by the student model in the previous iteration; Step 4) applying the named entity recognition method model based on the BERT model obtained after training in step 3) to the unstructured case text in the judicial field to automatically identify relevant entities; Step 3) builds the teacher-student framework including: 3-1) The model fine-tuned by the dataset is used as the initial teacher model, expressed as The model directly trained by the pre-trained model BERT-CNN without fine-tuning is initialized as the student model, which is expressed as 3-2) Named entity recognition task is the process of classifying named entities in text into predetermined entity types; formally given a sentence with N tokens, the annotation of the entity is related to the entity type; The teacher model proposed in 3-1) uses the formula Generate pseudo labels, perform iterative updates, and optimize them, where f n,c represents the probability that the nth token belongs to the cth category, X m Represented as the mth sentence; the student model uses the formula To fit these labels; 3-3) Using ADAM algorithm to optimize the formula And according to the formula The student model and the teacher model are iteratively updated; after multiple iterations, the entity category classified by the student model is used as the final entity category.

2. The method for named entity recognition in the judicial field based on the BERT model according to claim 1, characterized in that: in, In step 1), the remote supervision method is used to process the judicial field dataset, which includes the following processes: 1-1) Crawling a large amount of unstructured judicial text data from the Internet; 1-2) Segment the text data crawled in 1-1) into sentences and words, and then use named entity recognition tools to perform entity annotation on the segmented text data in units of sentences; 1-3) Using an external knowledge base as a supervision source, the entities marked in step 1-2) are matched with the concepts in the knowledge base, and relevant dictionary information is collected from multiple online resources to label the entities in the sentence.

3. The method for named entity recognition in the judicial field based on the BERT model according to claim 1, characterized in that: In step 2), fine-tuning the pre-trained model BERT-CNN means encoding the dataset annotated in step 1) and extracting the semantic features of the sentence using the pre-trained model BERT-CNN. Specifically, the following steps are included: 2-1) Input the labeled data set in step 1) into the model and convert it into an input vector form consisting of three word embeddings, i.e., the input form required by the BERT model; the input vector form consisting of three word embeddings, where: the first is to use the WordPiece model to convert the input sentence into a word vector form; the second is to use Segment Embedding to determine whether there is a contextual relationship between two sentences; the third is to obtain the word position information through Position Embedding; 2-2) Since the length of sentences input each time during training is inconsistent, when the length of the input sentence is less than the pre-defined maximum length, it is necessary to fill it with 0; the masked language model mechanism is used during training to randomly mask part of the words in the sentence and use the context of the instance to make predictions; finally, the input vector is input into the BERT model to obtain the low-dimensional vector representation corresponding to each sentence; 2-3) The low-dimensional vector processed by the BERT model is connected to the CNN network model to obtain the final semantic feature vector; wherein, the CNN network model includes three convolution modules and activation functions; the feature vector obtained in 2-2) is reduced in dimension through the pooling operation of the CNN network model, and then a linear layer and a dropout layer are passed to alleviate the overfitting of the network; during the training process of the CNN network model, an early stopping mechanism is also adopted, which is specifically manifested in that when the performance of the model on the validation set begins to decline, the training is stopped immediately to prevent the model from overfitting the data and improve the generalization ability of the data; 2-4) Use the BERT-CNN model fine-tuned in step 2-3) to train the entire dataset, use the softmax classifier to classify the processed semantic feature vectors, and predict a set of soft labels for all data.

4. A judicial named entity recognition system based on the BERT model, characterized in that: Includes the following modules: Dataset annotation module: select traffic accident verdicts in the judicial field as the data set, use remote supervision to process the data set, and annotate the data set; Pre-trained model BERT-CNN fine-tuning module: Part of the labeled data set is input into the pre-trained model BERT-CNN for fine-tuning. Early stopping measures are taken during the fine-tuning process, and a set of soft labels are predicted for all data using the BERT-CNN model; Teacher-student model training module: replace the dataset generated by the remote supervision method with the newly generated dataset with soft labels; design a teacher-student framework to further improve the recall rate by continuous training iteration; the student model is initialized by the model of the soft label dataset, and the teacher model is updated by the student model in the previous iteration; Entity recognition module: The named entity recognition method model based on the BERT model obtained after training in the teacher-student model training module is applied to unstructured case texts in the judicial field to automatically identify relevant entities; The teacher-student framework construction process is as follows: 3-1) The model fine-tuned by the dataset is used as the initial teacher model, expressed as The model directly trained by the pre-trained model BERT-CNN without fine-tuning is initialized as the student model, which is expressed as 3-2) Named entity recognition task is the process of classifying named entities in text into predetermined entity types; formally given a sentence with N tokens, the annotation of the entity is related to the entity type; The teacher model proposed in 3-1) uses the formula Generate pseudo labels, perform iterative updates, and optimize them, where f n,c represents the probability that the nth token belongs to the cth category, X m Represented as the mth sentence; the student model uses the formula To fit these labels; 3-3) Using ADAM algorithm to optimize the formula And according to the formula The student model and the teacher model are iteratively updated; after multiple iterations, the entity category classified by the student model is used as the final entity category.

5. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for named entity recognition in the judicial field based on the BERT model as described in any one of claims 1 to 3 is implemented.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the method for named entity recognition in the judicial field based on the BERT model as described in any one of claims 1 to 3.

Images