Chinese named entity recognition system, method and equipment based on multi-scale features and medium

Abstract

The invention discloses a Chinese named entity recognition system, method and equipment based on multi-scale features and a medium, and the method comprises the steps: decomposing an original text into a character sequence through a preprocessing module, and preprocessing characters, including removing punctuation marks and uniformly converting the characters into lower letters; a RoBERTa-WWM sub-module of a feature extraction module is responsible for converting an input text into a high-dimensional feature vector, a CNN sub-module effectively extracts local features through a sliding window mechanism, and a BiLSTM sub-module finally utilizes the advantage of bidirectional processing to obtain complete context information; decoding the hidden state representation by using a dynamic conditional random field (CRF) through a sequence tagging module to determine an optimal tag sequence; according to the method, pre-training, multi-scale feature fusion and dynamic decoding technologies are integrated, and efficient and accurate Chinese entity recognition is realized; through a unique preprocessing rule, a modular architecture design and optimized hyper-parameter configuration, the performance and robustness of the system in a complex scene are ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of text recognition, and in particular relates to a Chinese named entity recognition system, method, device and medium based on multi-scale features. Background Art

[0002] Named entity recognition (NER) is a core task in natural language processing (NLP), aiming to identify entities with specific meanings from text, such as person names, place names, organizations, time expressions, etc. NER is crucial in various application scenarios, such as information extraction, question answering systems, machine translation, sentiment analysis, image desensitization, etc.

[0003] With the development of the Internet, the amount of text data has increased exponentially. How to efficiently and accurately extract valuable information from massive text has become an important issue. As a basic step of information extraction, the performance of NER directly affects the effects of subsequent tasks.

[0004] Currently, the methods of named entity recognition are mainly divided into the following two categories: methods based on traditional machine learning and methods based on deep learning. Methods based on traditional machine learning use feature engineering to extract text features and use classifiers (such as support vector machine SVM, decision tree, maximum entropy model, etc.) for classification. However, traditional machine learning requires a large amount of professional knowledge to design effective features and also depends on the quality of the training set. When dealing with Chinese text, traditional models usually rely on a single network structure to extract features, resulting in the inability to fully utilize local character-level features and global context dependencies. For example, bidirectional long short-term memory network (BiLSTM) is good at capturing long-distance context dependencies, but is less sensitive to local character-level features; convolutional neural network (CNN) extracts local features through a sliding window, but its receptive field is fixed and it is difficult to model long-distance dependencies (literature "A Chinese Named Entity Recognition Method Based on BERT" by Wang Ziniu, 2019, "Computer Science").

[0005] In addition, the deep learning model (BiLSTM-CRF) is a classic combination of deep learning models for sequence annotation tasks. Although it can automatically learn text semantic information and capture long-distance context dependencies without manual feature design, it performs poorly in obtaining local character-level features and requires a large amount of computing resources. The transition matrix of the traditional CRF needs to preset the label transition probability in advance (for example, the probability of "B-PER" followed by "I-PER" is 0.9), but it cannot cover all possible context scenarios. For example, in the sentence "Zhang San is a professor at Peking University", the transition probability of "B-ORG" (Peking University) followed by "I-TITLE" (professor) may not be reasonably modeled, resulting in label conflicts. Experiments show that on the OntoNotes dataset, the label conflict rate of the traditional CRF is as high as 12% (such as in the Journal of Chongqing University of Posts and Telecommunications (Natural Science Edition), Vol. 31, No. 6, of the literature "Research on Chinese Electronic Medical Record Named Entity Recognition Based on CNN-CRF"). Summary of the Invention

[0006] To overcome the deficiencies of the above-mentioned prior art, the purpose of the present invention is to propose a Chinese named entity recognition system, method, device and medium based on multi-scale features. Aiming at the problem of Chinese named entity recognition, it integrates pre-training, multi-scale feature fusion and dynamic decoding technologies to achieve efficient and accurate Chinese entity recognition; through unique preprocessing rules, modular architecture design and optimized hyperparameter configuration, it ensures the performance and robustness of the system in complex scenarios.

[0007] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0008] A Chinese named entity recognition system based on multi-scale features, comprising:

[0009] Preprocessing module: decomposes the original text into a character sequence and preprocesses the characters, including removing punctuation marks and uniformly converting them to lowercase;

[0010] Feature extraction module: This module consists of a RoBERTa-WWM sub-module, a CNN sub-module and a BiLSTM sub-module; the RoBERTa-WWM sub-module is responsible for converting the input text into a high-dimensional feature vector; the CNN sub-module effectively extracts local features through a sliding window mechanism, and the BiLSTM sub-module finally utilizes the advantage of bidirectional processing to obtain complete context information;

[0011] Sequence annotation module: uses a dynamic conditional random field (CRF) to decode the hidden state representation to determine the optimal label sequence.

[0012] A Chinese named entity recognition method based on multi-scale features, specifically including the following steps:

[0013] Step 1: Use the preprocessing module to preprocess the input text, including removing noise, text normalization, and Chinese word segmentation, and output structured text;

[0014] Step 2: Use RoBERTa-WWM, CNN, and BiLSTM in the feature extraction module to fully obtain the context information in the structured text sequence data output in Step 1, and output hidden feature vectors;

[0015] Step 3: For the hidden feature vectors output in Step 2, use a dynamic conditional random field (CRF) to decode the hidden state representation and output the optimal label sequence.

[0016] The specific method of Step 1 is as follows:

[0017] Use the preprocessing module to first clean the original input text using regular expressions, including newline characters, blank lines, extra spaces, and irrelevant characters in the text; then use the OpenCC tool to uniformly convert the letters in the text to lowercase to reduce the interference caused by case differences and perform text normalization; finally, use the jieba word segmentation tool to segment the Chinese text, divide the continuous text into lexical units with independent meanings, and provide a high-quality text dataset for subsequent Chinese named entity recognition, that is, the structured text x = [x1, x2,..., x n .

[0018] The specific method of Step 2 is as follows:

[0019] 2.1 Train the RoBERTa-WWM sub-module. First, convert the preprocessed structured text x = [x1, x2,..., x n into an embedding vector E0 = [e1, e2,..., e n , and add position encoding P = [p1, p2,..., p3] to each character to generate an initial hidden state representation H0; then, through multiple layers of Transformer encoders, use the self-attention mechanism and the feed-forward neural network to capture context information and generate a hidden state representation H = [h1, h2,..., h n containing rich semantic features. The calculation formulas of the self-attention mechanism and the feed-forward neural network are shown in Equations (1) and (2) respectively:

[0020]

[0021] where, Q: query vector matrix, K: key vector matrix, V: value vector matrix, d k : dimension of the key vector;

[0022] FFN(H l ) = max(0, Hl W1 + b1)W2 + b2 (2)

[0023] Among them, H l is the hidden state representation of the l-th layer; W1 and b1 are the weight matrix and bias term of the first layer, and W1 and b2 are the weight matrix and bias term of the second layer;

[0024] The RoBERTa-WWM sub-module adopts a dynamic masking strategy during training. In each round of training, 15% of the characters in the input sequence are randomly selected for masking, and the whole-word masking strategy is used to mask the entire word, that is, when a certain character in a word is selected, all characters belonging to the same word are masked simultaneously to capture more complete semantic information; the cross-entropy loss calculation of the dynamic masking strategy and the whole-word masking strategy is shown in Formulas (3) and (4), and the total loss of the model obtained by weighted summation of Formulas (3) and (4) according to Formula (5) is calculated, and the Adam optimizer is used to update the model parameters to minimize the total loss of the model, so as to help the model capture fine-grained language features and overall semantic information. The word embedding matrix generated by the RoBERTa-WWM model is denoted as E ∈ R n×d , where n is the number of word segments of the input structured text, and d is the dimension of each word embedding vector;

[0025]

[0026] Among them, N mask is the number of masked characters, M is the index set of masked characters, V is the size of the vocabulary, and y ij is the one-hot encoded vector of the true character at the i-th position, is the probability distribution predicted by the model;

[0027]

[0028] Among them, M mask is the number of masked words, M is the index set of masked words, V is the size of the vocabulary, and y kj is the one-hot encoded vector of the true word at the k-th position, is the probability distribution predicted by the model;

[0029] TotalLoss = α·MLM + β·WWM (5)

[0030] Among them, α and β are hyperparameters used to adjust the weights of the MLM loss and the WWM loss;

[0031] 2.2 First, use the CNN sub-module to reorganize the word embedding matrix E ∈ R generated by RoBERTa-WWM n×d into a three-dimensional tensor H ∈ Rn×d×1 , where \(n\) represents the number of samples input to the CNN sub-module at one time, \(d\) represents the number of characters in each sample, and \(l\) represents the character embedding dimension and serves as the input to the CNN; then, one-dimensional convolutional kernels \(w\) of different widths are used k \(\in\mathbb{R}\) h×d to extract \(H\in\mathbb{R}\) n×d×1 local features, and the vector \(P\) that best reflects the lexical semantics and context information is retained through max-pooling operation i \(\in\mathbb{R}\) c , i.e., the max-pooling result of the \(i\)-th convolutional layer; \(P\) i \(\in\mathbb{R}\) c is concatenated to form a high-order feature vector \(V=(y_1,y_2,\cdots,y\) m ) of fixed length; the Spatial Dropout mechanism is introduced to perform structured dropout on the high-order feature vector \(V=(y_1,y_2,\cdots,y\) m ), that is, for each channel of the CNN, it is decided whether to discard the channel with a custom probability \(p\); finally, the high-order feature vector \(V'=(y_1,y_2,\cdots,y\) m ) extracted by the CNN feature enhancement layer is sent to the BiLSTM sub-module;

[0032] 2.3 Using the BiLSTM sub-module, the high-order feature vector \(V'=(y_1,y_2,\cdots,y\) m ) extracted by the CNN feature enhancement layer is used as the input sequence \(V\in\mathbb{R}\) m , and this sequence is processed in the forward order. For each time step \(t\), the forward LSTM will update the current hidden state \(h\) t and cell state \(c\) t+1 according to the current input \(y\) t+1 and the information (hidden state \(h\) t and cell state \(c\) t ) of all previous time steps to obtain the forward context information, and process \(V\in\mathbb{R}\) m in the backward reverse order in the same way to obtain the backward context information; then, the forward hidden state sequence is generated through the forward LSTM unit The backward hidden state sequence is generated through the backward LSTM unit Then these two sequences are concatenated at each time step to form the final hidden state representation \(h=(h_1,h_2,\cdots,h\) t ) containing the complete context information; finally, these hidden states are passed to the conditional random field (CRF) module to generate the optimal label sequence, thereby achieving accurate named entity recognition.

[0033] The specific method of step 3 is as follows:

[0034] First, the feature vector h = (h1, h2,..., h t ) of the feature extraction module is mapped to the label space by the sequence labeling module to obtain the unnormalized scores at each time step; a transition matrix A ∈ R k×k is defined, where A ij represents the score for transitioning from label i to label j; then, for a given label sequence y = (y1, y2,..., y n ), its path score score(y) is calculated according to formula (6);

[0035]

[0036] where: is the score assigned to label y t at time step t; is the score for transitioning from label y t -1 to label y t ;

[0037] Then, according to formula (7), the sum of the path scores of all possible label sequences, that is, the normalization factor Z(X), is calculated to ensure the validity of the probability distribution; finally, the optimal label sequence is found through the Viterbi algorithm;

[0038]

[0039] where y represents traversing all possible label sequences.

[0040] A Chinese named entity recognition device based on multi-scale features, comprising:

[0041] A memory for storing computer programs;

[0042] A processor for implementing the Chinese named entity recognition method based on multi-scale features described in steps 1 to 3 when executing the computer program.

[0043] A computer-readable storage medium storing a computer program, characterized in that the computer program can implement Chinese named entity recognition based on multi-scale features based on the method described in steps 1 to 3 when executed by a processor.

[0044] Compared with the prior art, the present invention has the following advantages:

[0045] 1. In terms of feature extraction, compared with traditional models that only rely on BiLSTM to extract global features and cannot capture character-level local features, the present invention uses multi-convolutions of the CNN sub-module to extract local features and uses the BiLSTM sub-module to obtain long-distance context dependencies.

[0046] 2. In terms of Chinese word segmentation dependence, Lattice LSTM relies on word segmentation tools and external word libraries. The word segmentation error rate (3%-5%) directly leads to the spread of entity recognition errors; the Chinese word segmentation coverage of spaCy and Stanza engines is incomplete, and they do not support custom dictionaries, resulting in poor domain adaptability. The present invention proposes to use character-level input, whole-word masking, and Chinese character radical embedding to enhance word-level modeling and improve the recognition ability of out-of-vocabulary words.

[0047] 3. In terms of dynamic label decoding, traditional CRF relies on manually setting a static transition matrix (such as the probability of "B-PER→I-PER" being fixed at 0.9), which cannot model complex label relationships; the input text mask of traditional models is statically generated in the preprocessing stage and remains unchanged during the training process; in addition, the static transition matrix requires re-tuning across domains and has poor migration ability. The present invention adopts the RoBERTa-WWM sub-module based on the dynamic masking strategy, which regenerates the mask for training samples in each training cycle. The model can obtain more character-level feature information from the same data. The dynamic masking strategy allows the same sentence to be masked and learned in different ways during different training epochs, thus increasing the diversity of training samples and contributing to named entity classification. The present invention also realizes the dynamic learning of label dependence relationships by introducing a self-attention dynamic transition matrix, significantly reducing the label conflict rate and improving the efficiency of cross-domain migration.

[0048] In summary, the present invention integrates pre-training, multi-scale feature fusion, and dynamic decoding technologies to achieve efficient and accurate Chinese entity recognition; through unique preprocessing rules, modular architecture design, and optimized hyperparameter configuration, the performance and robustness of the system in complex scenarios are ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is the system structure diagram of the present invention.

[0050] Figure 2 It is the flowchart of the preprocessing module of the present invention.

[0051] Figure 3 It is the flowchart of the sequence annotation module of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0052] The following further describes the present invention in detail with reference to the drawings and specific embodiments.

[0053] A Chinese named entity recognition system based on multi-scale features, comprising:

[0054] Preprocessing module: decomposes the original text into a character sequence and preprocesses the characters, including removing punctuation marks and uniformly converting to lowercase;

[0055] Feature extraction module: This module consists of the RoBERTa-WWM sub-module, the CNN sub-module, and the BiLSTM sub-module; the RoBERTa-WWM sub-module is responsible for converting the input text into high-dimensional feature vectors; the CNN sub-module effectively extracts local features through a sliding window mechanism, and the BiLSTM sub-module finally utilizes the advantages of bidirectional processing to obtain complete context information;

[0056] Sequence labeling module: Use a dynamic conditional random field (CRF) to decode the hidden state representation to determine the optimal label sequence.

[0057] A Chinese named entity recognition method based on multi-scale features specifically includes the following steps:

[0058] In this example, a total of 24,836 text data are obtained. The original text data may contain some irrelevant symbols, inconsistent capitalization, HTML tags, and other elements. The input sequence is the basis of the entire named entity recognition system, and its design and implementation directly affect the performance and effect of subsequent modules. Through reasonable text segmentation, embedding generation, position encoding, and preprocessing steps, the input layer can provide high-quality data for subsequent modules, thereby improving the accuracy and robustness of the entire system.

[0059] Step 1: Use the preprocessing module to preprocess the input text, including removing noise, text normalization, and Chinese word segmentation, and output structured text; the specific method is:

[0060] Use the preprocessing module to first clean the original input text using regular expressions, including line breaks, blank lines, extra spaces, and irrelevant characters in the text; then use the OpenCC tool to uniformly convert the letters in the text to lowercase to reduce the interference caused by case differences and perform text normalization; finally, use the jieba word segmentation tool to segment the Chinese text, dividing the continuous text into independent meaningful lexical units to provide a high-quality text data set for subsequent Chinese named entity recognition, that is, structured text x = [x1, x2,..., x n . The preprocessing module process is as Figure 2 shown.

[0061] Step 2: Use the RoBERTa-WWM, CNN, and BiLSTM of the feature extraction module to fully obtain the context information in the structured text sequence data output in Step 1 and output hidden feature vectors; the specific method is:

[0062] 2.1 Train the RoBERTa-WWM sub-module. First, convert the preprocessed structured text x = [x1, x2,..., x n into an embedding vector E0 = [e1, e2,..., e n], and add position code P = [p1, p2, ..., p3] to each character to generate the initial hidden state representation H0; then, through the multi-layer Transformer encoder, the self-attention mechanism and feedforward neural network are used to capture the context information to generate the hidden state representation H = [h1, h2, ..., h n ], the calculation formulas of the self-attention mechanism and feedforward neural network are formula (1) and formula (2) respectively:

[0063]

[0064] Where Q: query vector matrix, K: key vector matrix, V: value vector matrix, d k : the dimension of the key vector;

[0065] FFN(H l )=max(0,H l W1+b1)W2+b2 (2)

[0066] Among them, H l is the hidden state representation of the lth layer; W1 and b1 are the weight matrix and bias term of the first layer, W1 and b2 are the weight matrix and bias term of the second layer;

[0067] The RoBERTa-WWM submodule adopts a dynamic masking strategy during the training process. In each round of training, 15% of the characters in the input sequence are randomly selected for masking, and the whole word masking strategy is used to mask the entire word. That is, when a character in a word is selected, all characters belonging to the same word are masked at the same time. For example, when the character "北" in the word "北京大学" is selected, the entire word will be masked as "[MASK][MASK][MASK][MASK]" to capture more complete semantic information. The cross entropy loss calculation of the dynamic masking strategy and the whole word masking strategy is shown in formula (3) and formula (4). The total model loss of the weighted sum of formula (3) and formula (4) is calculated according to formula (5), and the Adam optimizer is used to update the model parameters to minimize the total model loss to help the model capture fine-grained language features and overall semantic information. The word embedding matrix generated after the RoBERTa-WWM model is represented as E∈R n×d , n is the number of word segments of the input structured text, and d is the dimension of each word embedding vector;

[0068]

[0069] Among them, N mask is the number of masked characters, M is the index set of masked characters, V is the size of the vocabulary, y ij is the one-hot encoding vector of the real character at position i, is the probability distribution predicted by the model;

[0070]

[0071] where M mask is the number of masked words, M is the index set of masked words, V is the size of the vocabulary, and y kj is the one-hot encoded vector of the true word at the k-th position, is the probability distribution predicted by the model;

[0072] TotalLoss = α·MLM + β·WWM (5)

[0073] where α and β are hyperparameters used to adjust the weights of the MLM loss and the WWM loss;

[0074] 2.2 First, use the CNN sub-module to reorganize the word embedding matrix E ∈ R n×d reorganized into a three-dimensional tensor H ∈ R n×d×1 , where n represents the number of samples input to the CNN sub-module at one time, d represents the number of characters per sample, and 1 represents the character embedding dimension and serves as the CNN input; then use one-dimensional convolutional kernels w k ∈ R h×d to extract local features of H ∈ R n×d×1 and retain the vector P i ∈ R c that best reflects the lexical semantics and context information through max-pooling operation, that is, the max-pooling result of the i-th convolutional layer; concatenate P i ∈ R c to form a high-order feature vector V of fixed length = (y1, y2,..., y m ); to further improve the generalization ability of the model, introduce the SpatialDropout mechanism to perform structured dropout on the high-order feature vector V = (y1, y2,..., y m ), that is, for each channel of the CNN, decide whether to discard the channel with a custom probability p; finally, obtain the high-order feature vector V' = (y1, y2,..., y m ) extracted by the CNN feature enhancement layer and send it to the BiLSTM sub-module;

[0075] 2.3 Use the BiLSTM sub-module to take the high-order feature vector V' = (y1, y2,..., y m ) extracted by the CNN feature enhancement layer as the input sequence V ∈ R m, process the sequence in forward order. For each time step t, the forward LSTM will, according to the current input y t and the information of all previous time steps (hidden state h t+1 and cell state c t+1 ), update the current hidden state h t and cell state c t to obtain the forward context information, and process V ∈ R in reverse order from the back to obtain the backward context information in the same way; then, generate a forward hidden state sequence through the forward LSTM cell m Generate a backward hidden state sequence through the backward LSTM cell Then concatenate these two sequences at each time step to form the final hidden state representation h = (h1, h2,..., h t ) containing the complete context information; finally, these hidden states are passed to the conditional random field (CRF) module to generate the optimal label sequence for accurate named entity recognition.

[0076] Step 3, for the hidden feature vectors output in Step 2, use the dynamic conditional random field (CRF) to decode the hidden state representation and output the optimal label sequence; the specific method is as follows:

[0077] The processing flow of the sequence annotation module is as Figure 3 shown. First, map the feature vector h = (h1, h2,..., h t ) of the feature extraction module to the label space to obtain the unnormalized score at each time step; to consider the dependencies between adjacent labels, define a transition matrix A ∈ R k×k where A ij represents the score for transitioning from label i to label j; then, for a given label sequence y = (y1, y2,..., y n ), calculate its path score score(y) according to formula (6);

[0078]

[0079] where: is the score assigned to label y t at time step t; is the score for transitioning from label y t -1 to label y t ;

[0080] Then, calculate the sum of the path scores of all possible label sequences, that is, the normalization factor Z(X), according to formula (7) to ensure the validity of the probability distribution; finally, find the optimal label sequence through the Viterbi algorithm;

[0081]

[0082] Among them, \(y\) represents traversing all possible tag sequences.

[0083] Experimental verification

[0084] The present invention was experimented on the text subset of the pre-desensitized image dataset that has been constructed, and the precision, recall, and F1 were used to comprehensively evaluate the Chinese named entity recognition results.

[0085] Experimental results of different Chinese named entity recognition models

[0086] The experimental results are shown in Table 1.1.

[0087] Table 1.1 Named entity recognition results of different models

[0088]

[0089] The F1 score of the model of the present invention is slightly lower than that of BERT-CRF, but it is superior to other methods in terms of accuracy and recall metrics. Comprehensive analysis shows that the Chinese named entity recognition model proposed by the present invention performs well in the NER task.

[0090] Ablation experiment results of the model of the present invention

[0091] To verify the innovation and rationality of the model of the present invention, ablation experiments were carried out on the text subset of the pre-desensitized image dataset. Taking the BiLSTM-CRF model as the benchmark, the CNN module and the RoBERTa-WWM module were added respectively to form comparative experimental groups: BiLSTM-CRF, CNN-BiLSTM-CRF, and RoBERTa-WWM-BiLSTM-CRF. By calculating the accuracy, recall, and F1 value metrics, the innovation and rationality of the model of the present invention were verified. The experimental results are shown in Table 1.2, and the accuracy, recall, and F1 value of the present invention are all significantly better than those of the comparative experimental groups.

[0092] Table 1.2 Ablation experiment results of the model of the present invention

[0093]

[0094] Through the above embodiments and experimental verifications, it is fully illustrated that:

[0095] This paper utilizes a parallel CNN-BiLSTM architecture and a gated fusion mechanism. Addressing the problem of traditional models extracting a single feature using a single network, this paper proposes using CNN and BiLSTM to independently process input features, extracting local character-level features (CNN) and global contextual dependencies (BiLSTM), respectively, to avoid the information attenuation associated with a serial architecture. A gated fusion mechanism dynamically adjusts the weights of local and global features, emphasizing local features to improve boundary recognition accuracy and enhance the ability to capture text information.

[0096] This paper adopts a Chinese-optimized pre-training strategy. It proposes a whole-word masking strategy and a Chinese character header embedding method. Unlike traditional character-level masking, WWM masks words at the unit level, thereby improving the model's overall understanding of words. By adding radical codes to each character (e.g., "病" → "疒"), the semantic representation of out-of-scope (OOV) words is enhanced, improving the model's understanding of out-of-scope (OOV) words.

[0097] This paper adopts a dynamic transfer matrix design and training method based on the self-attention mechanism. It proposes to transform the static transfer matrix A∈R of the traditional CRF into k×k The transfer matrix is replaced with a dynamic matrix A, and the transfer matrix is jointly optimized with the rest of the model, eliminating the need for manual initialization. This reduces the probability of label conflicts and significantly shortens the tuning time when migrating across domains.

[0098] This paper utilizes an end-to-end Chinese NER system. By integrating pre-training, multi-scale feature fusion, and dynamic decoding techniques, this system achieves efficient and accurate Chinese entity recognition. Its core features include unique preprocessing rules, a modular architecture, and optimized hyperparameter configuration, ensuring high performance and robustness in complex scenarios.

Claims

1. A Chinese named entity recognition system based on multi-scale features, characterized in that, Including: Preprocessing module: decomposes the original text into a character sequence, and preprocesses the characters, including removing punctuation marks and uniformly converting them to lowercase; Feature extraction module: This module consists of a RoBERTa-WWM sub-module, a CNN sub-module, and a BiLSTM sub-module; the RoBERTa-WWM sub-module is responsible for converting the input text into a high-dimensional feature vector; the CNN sub-module effectively extracts local features through a sliding window mechanism, and the BiLSTM sub-module finally utilizes the advantages of bidirectional processing to obtain complete context information; Sequence annotation module: uses a dynamic conditional random field (CRF) to decode the hidden state representation to determine the optimal label sequence.

2. A Chinese named entity recognition method based on multi-scale features, characterized in that, Specifically, it includes the following steps: Step 1: Use the preprocessing module to preprocess the input text, including removing noise, text normalization, and Chinese word segmentation, and output structured text; Step 2: Use the RoBERTa-WWM, CNN, and BiLSTM in the feature extraction module to fully obtain the context information in the structured text sequence data output in Step 1, and output a hidden feature vector; Step 3: For the hidden feature vector output in Step 2, use a dynamic conditional random field (CRF) to decode the hidden state representation and output the optimal label sequence.

3. A method for Chinese named entity recognition based on multi-scale features according to claim 2, characterized in that, The specific method of Step 1 is: The preprocessing module first uses regular expressions to clean the original input text, including newline characters, blank lines, extra spaces, and irrelevant characters in the text; then uses the OpenCC tool to uniformly convert the letters in the text to lowercase to reduce the interference caused by case differences and perform text normalization; finally, uses the jieba word segmentation tool to segment the Chinese text, dividing the continuous text into lexical units with independent meanings, providing a high-quality text dataset for subsequent Chinese named entity recognition, that is, the structured text x = [x1, x2,..., x n .

4. A method for Chinese named entity recognition based on multi-scale features according to claim 2, characterized in that The specific method of Step 2 is: 2.1 Train the RoBERTa-WWM sub-module. First, convert the pre-processed structured text x = [x1, x2,..., x n into the embedding vector E0 = [e1, e2,..., e n , and add the position encoding P = [p1, p2,..., p3] to each character to generate the initial hidden state representation H0; then, through multiple layers of Transformer encoders, use the self-attention mechanism and the feed-forward neural network to capture the context information and generate the hidden state representation H = [h1, h2,..., h n containing rich semantic features. The calculation formulas of the self-attention mechanism and the feed-forward neural network are formulas (1) and (2) respectively: Among them, Q: query vector matrix, K: key vector matrix, V: value vector matrix, d k : dimension of the key vector; FFN(H l ) = max(0, H l W1 + b1)W2 + b2 (2) where H l is the hidden state representation of the l-th layer; W1 and b1 are the weight matrix and bias term of the first layer, and W2 and b2 are the weight matrix and bias term of the second layer; The RoBERTa-WWM sub-module adopts a dynamic masking strategy during training. In each round of training, 15% of the characters in the input sequence are randomly selected for masking, and the whole-word masking strategy is used to mask the entire word. That is, when a certain character in a word is selected, all characters belonging to the same word are masked simultaneously to capture more complete semantic information. The cross-entropy loss calculations of the dynamic masking strategy and the whole-word masking strategy are shown in Formulas (3) and (4). The total loss of the model, which is the weighted sum of Formulas (3) and (4) according to Formula (5), is calculated, and the Adam optimizer is used to update the model parameters to minimize the total loss of the model, so as to help the model capture fine-grained language features and overall semantic information. The word embedding matrix generated by the RoBERTa-WWM model is denoted as E ∈ R n×d , where n is the number of word segments of the input structured text, and d is the dimension of each word embedding vector; Among them, N mask is the number of masked characters, M is the index set of masked characters, V is the size of the vocabulary, y ij is the one-hot encoded vector of the true character at the i-th position, is the probability distribution predicted by the model; Among them, M mask is the number of masked words, M is the index set of masked words, V is the size of the vocabulary, y kj is the one-hot encoded vector of the true word at the k-th position, is the probability distribution predicted by the model; TotalLoss = α·MLM + β·WWM (5) Where α and β are hyperparameters used to adjust the weights of the MLM loss and the WWM loss; 2.2 First, use the CNN sub-module to re-organize the word embedding matrix E ∈ R n×d generated by RoBERTa-WWM into a three-dimensional tensor H ∈ R n×d×1 , where n represents the number of samples input to the CNN sub-module at one time, d represents the number of characters in each sample, and 1 represents the character embedding dimension and serves as the CNN input; then use one-dimensional convolutional kernels w of different widths k ∈ R h×d to extract local features of H ∈ R n×d×1 , and retain the vector P i ∈ R c that best reflects the lexical semantics and context information through max-pooling operation, that is, the max-pooling result of the i-th convolutional layer; splice P i ∈ R c to form a high-order feature vector V of fixed length = (y1, y2,..., y m ); introduce the Spatial Dropout mechanism to perform structured dropout on the high-order feature vector V = (y1, y2,..., y m ), that is, for each channel of the CNN, decide whether to discard the channel with a custom probability p; finally, obtain the high-order feature vector V' = (y1, y2,..., y m ) extracted by the CNN feature enhancement layer and send it to the BiLSTM sub-module; 2.3 Use the BiLSTM sub-module to take the high-order feature vector V'=(y1, y2,..., y m ) extracted by the CNN feature enhancement layer as the input sequence V∈R m . Process this sequence in the forward order. For each time step t, the forward LSTM will, according to the current input y t and the information of all previous time steps (hidden state h t+1 and cell state c t+1 ), update the current hidden state h t and cell state c t to obtain the forward context information, and process V∈R m in the reverse order from the back to obtain the backward context information; then, generate the forward hidden state sequence through the forward LSTM unit generate the backward hidden state sequence through the backward LSTM unit Then concatenate these two sequences at each time step to form the final hidden state representation h=(h1, h2,..., h t ) containing the complete context information; finally, these hidden states are passed to the conditional random field (CRF) module to generate the optimal label sequence, thereby achieving accurate named entity recognition.

5. A method for Chinese named entity recognition based on multi-scale features according to claim 2, characterized in that The specific method of Step 3 is: Through the sequence labeling module, the feature vector h=(h1, h2,..., h t ) of the feature extraction module is first mapped to the label space to obtain the unnormalized scores at each time step; a transition matrix A∈R k×k is defined, where A ij represents the score for transitioning from label i to label j; then, for a given label sequence y=(y1, y2,..., y n ), its path score score(y) is calculated according to formula (6); Where: is the score assigned to label y at time step t t ; is the score for transitioning from label y t -1 to label y t ; Then, calculate the sum of the path scores of all possible label sequences, that is, the normalization factor Z(X), according to formula (7) to ensure the effectiveness of the probability distribution; finally, find the optimal label sequence through the Viterbi algorithm; Where y' represents traversing all possible label sequences.

6. A Chinese named entity recognition device based on multi-scale features, characterized in that, Including: A memory for storing a computer program; A processor for implementing the Chinese named entity recognition method based on multi-scale features described in claims 1 to 5 when executing the computer program.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can realize Chinese named entity recognition based on multi-scale features according to the method described in claims 1 to 5.

Images