A Chinese judicial field named entity recognition method fusing multiple external semantic features

Abstract

This invention discloses a method for Chinese legal domain named entity recognition that integrates multiple external semantic features, comprising the following steps: 1) constructing a legal corpus; 2) constructing an undirected glyph graph at the global level of the dataset, and using the glyph graph to enhance the glyphs of the input data of the model embedding layer; 3) performing multi-vocabulary enhancement on the input data of the model embedding layer; 4) fusing the inputs enhanced by the two external semantic features obtained in steps 2) and 3) through a gating mechanism, and then inputting them into a BiLSTM-CRF network to recognize named entities in the legal domain. This method can effectively recognize seven types of entities in legal texts: crime scene, place names, personal names, organization names, crimes, legal provisions, and sentences. By enhancing with multiple external semantic features, it reduces the dependence on legally annotated corpora, and improves the recognition accuracy and F1 score for limited legally annotated corpora.

Description

Technical Field

[0001] This invention belongs to the field of named entity recognition in natural language processing, and specifically relates to named entity recognition in the Chinese judicial field. It is a method for named entity recognition in the Chinese judicial field that integrates multiple external semantic features. Background Technology

[0002] When people access information, search for content, and interact with information on the internet, they generate a large amount of text data. To improve the processing speed of Chinese text, many applications in the field of Chinese natural language processing require the support of Chinese named entity recognition, including tasks such as intelligent recommendation, question answering systems, text understanding, and text generation.

[0003] The main task of Chinese Named Entity Recognition (NER) is to identify entities from raw text and classify the detected entities into predefined categories, such as "person names," "place names," "organizations," and other proper nouns. For NER, the mainstream approach is to build vocabulary-based models. These models combine external lexical information to semantically enhance the input text, thereby identifying entity types within the text. The most typical method is Lattice LSTM (Zhang, Y.; Yang, J. Chinese NER Using Lattice LSTM. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Melbourne, Australia, 15–20 July 2018; pp. 1554–1564.), which inputs lexical knowledge into the state of the LSTM network, thus constructing a raster-like structure to introduce lexical knowledge into the Chinese NER model. However, this method is limited by its construction method; the lexical representation can only be added to the last word, and the raster structure cannot be computed in parallel. To address this issue, Sui et al. proposed a lexical knowledge-based approach using collaborative graphs (Sui, D.; Chen, Y.; Liu, K.; Zhao, J.; Liu, S. LeverageLexical Knowledge for Chinese Named Entity Recognition via CollaborativeGraph Network. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9 thThe International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, 3–7 November 2019; pp. 3830–3840., utilizes large-scale automatic text segmentation for pre-training and graph convolution to achieve automatic vocabulary construction. In the same year, Ding et al. proposed a named entity recognition method based on graph neural networks and combined with multiple dictionaries (Ding, R.; Xie, P.; Zhang, X.; Lu, W.; Li, L.; Si, L. A Neural Multi-digraph Model for Chinese NER with Gazetteers. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, 28 July–2 August 2019; pp. 1462–1467.), enabling the model to automatically learn dictionary features, which helps alleviate the problem of false matching.

[0004] These methods all require the construction of dictionary resources, and the quality of the dictionary directly affects the recognition performance of named entities. Another typical method is the Glyce network proposed by Meng et al. (Meng, Y.; Wu, W.; Wang, F.; Li, X.; Nie, P.; Yin, F.; Li, M.; Han, Q.; Sun, X.; Li, J. Glyce: Glyph-vectors for Chinese Character Representations. In Proceedings of the Advances in Neural Information Processing Systems; Curran Associates, Inc.: Red Hook, NY, USA, 2019; Volume 32.). This network extracts features from Chinese character images using a CNN network to obtain the glyph information of the characters, and then fuses the glyph information and character information through image classification to finally perform named entity recognition. Similarly, Wu et al. (Wu, S.; Song, X.; Feng, Z. MECT: Multi-Metadata Embedding based Cross-Transformer for Chinese Named Entity Recognition. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Virtual Event, 1–6 August 2021; Association for Computational Linguistics: Stroudsburg, PA, USA, 2021; pp. 1529–1539.) extracted the radicals of Chinese characters using CNNs, and then incorporated the radical and character information into the model through a Cross-Transformer module and a stochastic attention mechanism to improve the accuracy of Chinese named entity recognition. However, these models mainly extract the glyph information of individual Chinese characters through CNNs, neglecting the interaction knowledge between glyphs and context.

[0005] While the aforementioned work achieved high accuracy in Chinese named entity recognition, its performance remained poor on datasets with complex contexts, and the model structure was also quite complex. This paper proposes a model based on the general BiLSTM-CRF network structure. It introduces glyph information such as Chinese character radicals and semantic knowledge from various word segmentation methods through a loosely coupled adaptive embedding layer, thereby enhancing the input with glyph and vocabulary, thus improving the accuracy of Chinese named entity recognition. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a Chinese legal domain named entity recognition method that integrates multiple external semantic features. This method constructs a legally annotated corpus and enhances the input of the named entity recognition model through various external semantic features. It achieves high F1 scores and recognition accuracy in recognizing seven types of entities: "crime location, place name, personal name, organization name, crime, legal provision, and sentence."

[0007] The technical solution to achieve the objective of this invention is:

[0008] A method for named entity recognition in the Chinese judicial domain that integrates multiple external semantic features includes the following steps:

[0009] 1) Construct a judicial corpus. First, the original data is sequence-labeled. The labeled judicial corpus dataset is then divided into a training set, a validation set, and a test set in a 6:2:2 ratio.

[0010] 2) Construct an undirected glyph graph at the global level of the dataset, and use the glyph graph to enhance the glyphs of the input data of the model's embedding layer;

[0011] 3) Perform multi-vocabulary augmentation on the input data of the model's embedding layer;

[0012] 4) The inputs enhanced by the two external semantic features obtained in steps 2) and 3) are fused through a gating mechanism and then input into the BiLSTM-CRF network to identify named entities in the judicial field.

[0013] Furthermore, in step 1), the original data consists of publicly available judicial judgment documents. By constructing a dynamic dictionary, the seven entity types of the judicial judgment documents are sequence-labeled to obtain a labeled judicial corpus. The entity types are: crime location, place name, personal name, organization name, crime, legal provision, and sentence. The labels are in BIO format. The specific labeling method is as follows: for each character, M is defined as the label of its entity type, then BM represents the beginning of the named entity, IM represents the rest of the named entity, and O represents non-named entity characters.

[0014] Furthermore, the method for enhancing the glyphs in step 2) is as follows:

[0015] 2.1) Construct a dictionary of Chinese character decompositions, in the format: "character + radical 1 + radical 2...";

[0016] 2.2) Map each character and radical in the dataset to a one-hot vector as the initial node representation of the glyph graph, thereby constructing a global feature matrix at the dataset level. ,in The number of all distinct characters and radicals in the dataset;

[0017] 2.3) Construct an adjacency matrix to represent the edges of the glyph graph, using a Chinese character decomposition dictionary and a length of... A sliding window is used to perform co-occurrence statistics on the dataset, and the weights between different nodes are calculated using formula (1).

[0018]

[0019] In formula (1) This is mutual information between nodes, reflecting the information between two nodes. The degree of correlation between them, when When, it indicates that there is a strong semantic relationship between the two characters. When the weight is greater than 1, it indicates that there is almost no semantic relevance between the two characters, and finally, when the weight is greater than 1, it means that there is almost no semantic relevance between the two characters. Adding edges between nodes, we get the set of all edges as follows: ,in For nodes The edge between, The set of edges formed by all nodes;

[0020] 2.4) Calculate the importance scores of other nodes for the character using a dynamic attention mechanism. First, calculate the importance scores of neighboring nodes using formula (2). Features for nodes Importance score ,

[0021]

[0022] in, and These are learnable parameters. and They are nodes The character representation, The loss function is then used, and the importance scores are normalized using formula (3) to calculate the neighboring nodes. Features for nodes Attention score ,

[0023]

[0024] in, For nodes The set of all neighboring nodes is then weighted and summed using formula (4) and layer normalized to obtain the final node. The radical indicates ,

[0025] .

[0026] Furthermore, the multi-vocabulary enhancement steps in step 3) are as follows:

[0027] 3.1) Use three different tokenizers to process the input text sequence. Perform word segmentation, where For the character, three different segmentation results were obtained:

[0028]

[0029]

[0030]

[0031] in The segmented potential words are all the segmentation results that constitute a potential word set. ;

[0032] 3.2) For each character in the text sequence Match all relevant potential words Based on the character's position within the potential words, all potential words are divided into four categories. This is used to represent the location information of the potential word, i.e. , , , ;

[0033] 3.3) Calculate the importance score of each character in the text sequence using formulas (8) and (9).

[0034]

[0035]

[0036] in For attention matrix, For pre-trained character-level embeddings, For the input sequence The index matrix will Attention representation of each character obtained by dividing by column. ,in For the first Attention scores for each character, based on segmentation results obtained from multiple tokenizers, for latent words. By combining their attention scores, we obtain Since different potential words have different lengths, formula (10) is used to aggregate potential words of different lengths into a unified dimension attention representation.

[0037]

[0038] in as potential words Attention aggregation It is the length of the potential word. For the maximum pool, For the average pool, It is a trainable weight parameter used to balance the max pool and the average pool;

[0039] 3.4) Using a length of The sliding window is used to count the probability of each latent word appearing in the training set. As the weight of each latent word, the attention representations of all words in the same category are weighted and summed using formula (11) to obtain the feature vector of that category.

[0040]

[0041] in Finally, the feature vectors of the four categories are concatenated into the character representation of each character using formula (12):

[0042] .

[0043] Furthermore, the feature fusion method in step 4) is as follows:

[0044] 4.1) The contribution is obtained by performing a linear transformation on the character representations after glyph enhancement and lexical enhancement using formula (13) and then concatenating them. ,

[0045]

[0046] in, The input sequence after vocabulary enhancement. The input sequence after glyph enhancement. The sigmoid function is used, and then the gating mechanism shown in formula (14) is used to dynamically weight the lexical features and glyph features to obtain the character-level representation after feature fusion. ,

[0047]

[0048] in, It is the output of the gating mechanism. It is a matrix consisting entirely of 1s. This is an element-wise multiplication operation. This is a vector join operation; finally, When input into a BiLSTM-CRF network, the label sequence corresponding to the highest probability value during the recognition process is the optimal named entity recognition result.

[0049] Compared with existing methods, the method of the present invention has the following advantages:

[0050] 1. This invention annotates judicial judgment documents by constructing a dynamic dictionary, which reduces the manual and time costs of manually annotating data and is conducive to the expansion of the judicial corpus.

[0051] 2. The glyph enhancement in this invention can improve the model's reasoning ability on category information and enhance the recognition ability of "crime scene" entity category, which has a complex structure and low frequency, in judicial texts.

[0052] 3. This invention enhances vocabulary by introducing various lexical information, thereby improving the ability to identify a large number of similar "legal provisions" and "crimes" entity categories in judicial texts.

[0053] 4. This invention enhances the judicial annotation corpus by using a variety of external semantic features, thereby reducing the reliance on the judicial annotation corpus and improving the recognition accuracy and F1 score of named entities in the judicial field for the limited judicial annotation corpus. Attached Figure Description

[0054] Figure 1 A flowchart of the identification method for the embodiment;

[0055] Figure 2 This is a flowchart of the character enhancement process in the character recognition method of the embodiment;

[0056] Figure 3 This is a flowchart of the vocabulary enhancement process in the identification method of the embodiment. Detailed Implementation

[0057] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments, but this is not intended to limit the scope of the invention.

[0058] Example:

[0059] Reference Figure 1 A method for named entity recognition in the Chinese judicial domain that integrates multiple external semantic features includes the following steps:

[0060] 1) Construct a judicial corpus. First, the original data is sequence-labeled. The labeled judicial corpus dataset is then divided into a training set, a validation set, and a test set in a 6:2:2 ratio.

[0061] 2) Construct an undirected glyph graph at the global level of the dataset, and use the glyph graph to enhance the glyphs of the input data of the model's embedding layer;

[0062] 3) Perform multi-vocabulary augmentation on the input data of the model's embedding layer;

[0063] 4) The inputs enhanced by the two external semantic features obtained in steps 2) and 3) are fused through a gating mechanism and then input into the BiLSTM-CRF network to identify named entities in the judicial field.

[0064] In this embodiment, the raw data used in step 1) comes from two sources: publicly available judicial judgments from the "China Judgments Online" website and the corpus published by the China "Legal Research Cup" Artificial Intelligence Competition, encompassing criminal, civil, and administrative cases. The specific method for constructing the dataset is as follows:

[0065] 1.1) Perform regularization extraction and de-identification preprocessing on the original data;

[0066] 1.2) Construct a dynamic dictionary and perform sequence annotation on seven entity types in judicial judgment documents to obtain an annotated judicial corpus. The entity types are: crime scene, place name, personal name, organization name, crime, legal provision, and sentence. The annotation labels are in BIO format. The specific annotation method is as follows: for each character, define M as the label of its entity type, then BM represents the beginning of a named entity, IM represents the rest of the named entity, and O represents a non-named entity character.

[0067] In this embodiment, the method for character enhancement in step 2) is as follows:

[0068] 2.1) Construct a Chinese character decomposition dictionary using the online Xinhua Dictionary, in the format: "character + radical 1 + radical 2...", with a total of 73 radicals;

[0069] 2.2) Map each character and radical in the dataset into a one-hot vector. As the initial node of the glyph graph, it is used to construct a global feature matrix at the dataset level. ,in The number of all distinct characters and radicals in the dataset. The matrix is ​​initialized to an identity matrix;

[0070] 2.3) Construct an adjacency matrix to represent the edges of the glyph graph, using a Chinese character decomposition dictionary and length. A sliding window is used to perform co-occurrence statistics on the dataset, and the weights between different nodes are calculated using formula (1).

[0071]

[0072] In formula (1) Inter-node mutual information reflects the degree of association between two nodes. At the node and nodes Add edges between them to get the set of all edges: ,in For nodes The edge between, It is the set of edges formed by all nodes.

[0073] 2.4) A set of text sequences to be recognized The input is fed into the embedding layer of the model, and the importance score of other nodes for the character is calculated using a dynamic attention mechanism. First, the neighboring nodes are calculated using formula (2). Features for nodes Importance score ,

[0074]

[0075] in, and These are learnable parameters. and They are nodes The character representation, The loss function is then used, and the importance scores are normalized using formula (3) to calculate the neighboring nodes. Features for nodes Attention score

[0076]

[0077] in For nodes The set of all neighboring nodes is then weighted and summed using formula (4) and layer normalized to obtain the final node. The radical indicates ,

[0078] .

[0079] In this embodiment, the multi-vocabulary enhancement step in step 3) is as follows:

[0080] 3.1) A set of text sequences to be recognized The input is fed into the model's embedding layer, where For each character, three different tokenizers are used for word segmentation, resulting in three different sets of segmentation results:

[0081]

[0082]

[0083]

[0084] in The segmented potential words are all the segmentation results that constitute a potential word set. ;

[0085] 3.2) For each character in the text sequence Match all relevant potential words Based on the character's position within the potential word, the potential words are divided into four categories. This is used to represent the location information of the potential word, i.e. , , , ;

[0086] 3.3) Calculate the importance score of each character in the text sequence using formulas (8) and (9).

[0087]

[0088]

[0089] in, For attention matrix, For pre-trained character-level embeddings, For the input sequence The index matrix will Attention representation of each character obtained by dividing by column. ,in For the first Attention scores for each character, based on segmentation results obtained from multiple tokenizers, for latent words. It is obtained by integrating their attention scores. Since different potential words have different lengths, formula (10) is used to aggregate potential words of different lengths into a unified dimension attention representation.

[0090]

[0091] in, as potential words Attention aggregation It is the length of the potential word. For the maximum pool, For the average pool, It is a trainable weight parameter used to balance the max pool and the average pool;

[0092] 3.4) Using a length of The sliding window is used to count the probability of each latent word appearing in the training set. As the weight of each latent word, the attention representations of all words in the same category are weighted and summed using formula (11) to obtain the feature vector of that category.

[0093]

[0094] in, Finally, the feature vectors of the four categories are concatenated into the character representation of each character using formula (12):

[0095] .

[0096] In this embodiment, the feature fusion method in step 4) is as follows:

[0097] 4.1) The contribution R is obtained by performing a linear transformation on the character representations after glyph enhancement and lexical enhancement using formula (13) and then concatenating them.

[0098]

[0099] in, The input sequence after vocabulary enhancement. The input sequence after glyph enhancement. The sigmoid function is used, and then the gating mechanism shown in formula (14) is used to dynamically weight the lexical features and glyph features to obtain the character-level representation after feature fusion.

[0100]

[0101] in It is the output of the gating mechanism. It is a matrix consisting entirely of 1s. This is an element-wise multiplication operation. This is a vector join operation; finally, The input is fed into a BiLSTM-CRF network. During the recognition process, the label sequence corresponding to the highest probability value is the optimal named entity recognition result, and the final output label sequence is obtained. ,in , Input text respectively The corresponding label results.

Claims

1. A method for named entity recognition in the Chinese legal field that integrates multiple external semantic features, characterized in that, Includes the following steps: 1) Construct a judicial corpus. First, the original data is sequence-labeled. The labeled judicial corpus dataset is then divided into a training set, a validation set, and a test set in a 6:2:2 ratio. The original data consists of publicly available judicial judgments. By constructing a dynamic dictionary, the seven entity types of the judicial judgments are sequentially labeled to obtain a labeled judicial corpus. The entity types are: crime location, place name, personal name, organization name, crime, legal provision, and sentence. The labels are in BIO format. The specific labeling method is as follows: for each character, M is defined as the label of its entity type, BM represents the beginning of a named entity, IM represents the rest of the named entity, and O represents a non-named entity character. 2) Construct an undirected glyph graph at the global level of the dataset, and use the glyph graph to enhance the glyphs of the input data of the model's embedding layer; The method for enhancing the character shape is as follows: 2.1) Construct a dictionary of Chinese character decompositions, in the format: "character + radical1 + radical2..."; 2.2) Map each character and radical in the dataset to a one-hot vector as the initial node representation of the glyph graph, thereby constructing a global feature matrix at the dataset level. ,in The number of all distinct characters and radicals in the dataset; 2.3) Construct an adjacency matrix to represent the edges of the glyph graph, using a Chinese character decomposition dictionary and a length of... A sliding window is used to perform co-occurrence statistics on the dataset, and the weights between different nodes are calculated using formula (1). In formula (1) This is mutual information between nodes, reflecting the information between two nodes. The degree of correlation between them, when When, it indicates that there is a strong semantic relationship between the two characters. When the weight is greater than 1, it indicates that there is almost no semantic relevance between the two characters, and finally, when the weight is greater than 1, it means that there is almost no semantic relevance between the two characters. Adding edges between nodes, we get the set of all edges as follows: ,in For nodes The edge between, The set of edges formed by all nodes; 2.4) Calculate the importance scores of other nodes for the character using a dynamic attention mechanism. First, calculate the importance scores of neighboring nodes using formula (2). Features for nodes Importance score , in, and These are learnable parameters. and They are nodes The character representation, The loss function is then used, and the importance scores are normalized using formula (3) to calculate the neighboring nodes. Features for nodes Attention score , in For nodes The set of all neighboring nodes is then weighted and summed using formula (4) and layer normalized to obtain the final node. The radical indicates , 3) Perform multi-vocabulary augmentation on the input data of the model's embedding layer; 4) The inputs enhanced by the two external semantic features obtained in steps 2) and 3) are fused through a gating mechanism and then input into the BiLSTM-CRF network to identify named entities in the judicial field.

2. The Chinese legal domain named entity recognition method according to claim 1, characterized in that: Step 3) involves multiple vocabulary enhancement steps: 3.1) Use three different tokenizers to process the input text sequence. Perform word segmentation, where For the character, three different segmentation results were obtained: in The segmented potential words are all the segmentation results that constitute a potential word set. ; 3.2) For each character in the text sequence Match all relevant potential words Based on the character's position within the potential words, all potential words are divided into four categories. This is used to represent the location information of the potential word, i.e. , , , ; 3.3) Calculate the importance score of each character in the text sequence using formulas (8) and (9). in For attention matrix, For pre-trained character-level embeddings, For the input sequence The index matrix will Attention representation of each character obtained by dividing by column. ,in For the first Attention scores for each character, based on segmentation results obtained from multiple tokenizers, for latent words. By combining their attention scores, we obtain Since different potential words have different lengths, formula (10) is used to aggregate potential words of different lengths into a unified dimension attention representation. in, as potential words Attention aggregation It is the length of the potential word. For the maximum pool, For the average pool, It is a trainable weight parameter used to balance the max pool and the average pool; 3.4) Using a length of The sliding window is used to count the probability of each latent word appearing in the training set. As the weight of each latent word, the attention representations of all words in the same category are weighted and summed using formula (11) to obtain the feature vector of that category. in, Finally, the feature vectors of the four categories are concatenated into the character representation of each character using formula (12): 。 3. The Chinese legal domain named entity recognition method according to claim 2, characterized in that: The feature fusion method in step 4) is as follows: The contribution is obtained by performing a linear transformation on the character representations after glyph enhancement and lexical enhancement using formula (13) and then concatenating them. , in, The input sequence after vocabulary enhancement. The input sequence after glyph enhancement. The sigmoid function is used, and then the gating mechanism shown in formula (14) is used to dynamically weight the lexical features and glyph features to obtain the character-level representation after feature fusion. , in, It is the output of the gating mechanism. It is a matrix consisting entirely of 1s. This is an element-wise multiplication operation. This is a vector join operation; Finally When input into a BiLSTM-CRF network, the label sequence corresponding to the highest probability value during the recognition process is the optimal named entity recognition result.

Images