A method for applying deep learning to Chinese word segmentation

Abstract

This invention belongs to the field of computer language processing technology, specifically a method for applying deep learning to Chinese word segmentation. Addressing the issues of high complexity, slow segmentation speed, and susceptibility to long-term dependencies when using recurrent neural networks for Chinese word segmentation, this invention proposes the following solution, including the following steps: first, data preprocessing; second, constructing a dictionary for model input; and finally, training the model. Data preprocessing includes segmenting the document using existing punctuation marks, paragraph separators, and other segmentation symbols, and extracting sentence character features. This invention employs a combination of Universal-Transformer and Conditional Random Field (BPEmd) methods for Chinese word segmentation. By fully utilizing the advantages of various algorithms and continuously updating the model parameters through a large training corpus, relying on the model's powerful learning ability, the Chinese word segmentation results are more accurate.

Description

Technical Field

[0001] This invention relates to the field of computer language processing technology, and in particular to a method for applying deep learning to Chinese word segmentation. Background Technology

[0002] Natural Language Processing (NLP) is one of the most important technologies in the information age, and Chinese word segmentation is the process of dividing a sequence of Chinese characters into individual words. It is the foundation of NLP and has a wide range of applications, such as part-of-speech tagging, named entity recognition, speech synthesis, automatic classification, and machine translation. In these tasks, the finest granularity is at the word level, words form sentences, sentences form paragraphs, and articles form articles. Traditional word segmentation methods are based on statistics. The advantages of these methods are simplicity, ease of implementation, and generally high efficiency, but their accuracy is not high, and they cannot solve ambiguity problems well, especially for professional documents in specific fields. Deep learning-based word segmentation methods have strong generalization ability, do not rely on dictionaries, and are trained on large-scale corpora. Their accuracy has been greatly improved compared to traditional word segmentation methods, but the algorithm complexity is high, and they are prone to gradient vanishing or gradient exploding problems. At the same time, they are heavily dependent on the selected corpus, which can lead to overfitting.

[0003] Currently, deep learning Chinese word segmentation algorithms are mainly based on recurrent neural network models, represented by Long Short-Term Memory (LSTM) and their derivative models. However, the computation of recurrent neural networks is sequential, meaning that related algorithms can only compute sequentially from left to right or from right to left. This mechanism brings two problems: the computation of time slice t depends on the computation result at time t-1, which limits the parallel computing capability of the model; information is lost during sequential computation. Although the structure of gate mechanisms such as LSTM alleviates the long-term dependency problem to some extent, LSTM is still powerless for particularly long-term dependency phenomena. Summary of the Invention

[0004] Chinese word segmentation based on recurrent neural networks is complex, slow, and prone to long-term dependency issues. This invention proposes a method that applies deep learning to Chinese word segmentation.

[0005] This invention proposes a method for applying deep learning to Chinese word segmentation, comprising the following steps:

[0006] S1: Data preprocessing: The document is segmented using existing punctuation marks, paragraph separators, and other symbols that have a segmenting function. The word features of sentences are extracted to obtain the first data.

[0007] S2: Construct a dictionary for model input, which consists of two parts: a source dictionary src_dict and a target dictionary tgt_dict. Further process the first data to construct the dictionary for model input and obtain the second data.

[0008] S3: Use the model for training. Input src_dict and tgt_dict into the training model. Use the BPEmd model to perform word embedding pre-training on the preprocessed data. The first data is used as the input of Universal-Transformer to obtain the feature vector after the dimension is increased. The second data is used as the input of CRF layer to obtain the target word vector model.

[0009] Preferably, the character features in S1 include four word positions: the first character of the word (B), the middle character of the word (M), the last character of the word (E), and the single character forming a word (S). The target corpus data is labeled by word position to obtain the first data, and the corpus data is converted into JSON format. In order to accelerate training, the first data is converted into HDF5 format.

[0010] Preferably, the corpus data set is divided into sentence sequences using spaces, and then the sentence sequences are divided into two sequences using tabs "\t" and sequences containing single characters and word position labels corresponding to the characters. The two sequences are traversed, and the frequency of each character is counted. If the frequency is less than the threshold of 5, it is ignored. Finally, the source dictionary src_dict and the target dictionary tgt_dict are obtained.

[0011] Preferably, in step S3, a model is used for training, and the trained model includes:

[0012] S31: Use the BPEmd model to perform character embedding pre-training on the preprocessed data, which increases the character feature dimension of the character vectors, and obtains the feature vectors with increased dimension.

[0013] S32: The first data in S1 is used as the input of Universal-Transformer. The model encodes a certain character and performs operations with other characters in the sequence to obtain the syntactic and semantic relationship encoding between the character and the characters to the left and right in the sequence, thus obtaining a vector that extracts the contextual relationship features of the character.

[0014] S33: The second data in S1 is used as the input of the CRF layer. The probability of classifying various word positions of each character vector in the corpus data is revised according to the context information to obtain the target character vector model.

[0015] Preferably, a sufficiently large training corpus dataset is prepared, the desired vocabulary size is determined, words are split into character sequences and a suffix "" is added to the end, word frequencies are counted, the frequency of each consecutive byte pair is counted, the highest frequency pair is selected and merged into a new one, and the aforementioned steps are repeated until the set vocabulary size is reached or the frequency of the next highest frequency byte pair is 1. The stop character "" represents a word suffix, and the frequency increases with the number of merging operations.

[0016] Preferably, in S32, the Universal-Transformer model takes a character sequence as input, and after the encoder and decoder extract the semantic and syntactic features between the characters, it outputs another character sequence according to the task type. The sentence input from the encoder first passes through a self-attention layer, which helps the encoder pay attention to other characters in the input sequence when encoding each character. The output of the self-attention layer is passed to the feedforward neural network, which is a transition layer that is cyclically calculated using a shared weight method. Here, "cyclic" refers to a deep loop, where the feedforward neural network corresponding to each word position is exactly the same. The output of the encoder serves as the input to the decoder, which also contains the encoder's self-attention layer and feedforward layer. Between these two layers, there is another attention layer used to focus on the relevant parts of the input sequence. For example, a sequence (a,b,c,d) is first represented by embedding as ( , After passing through a layer of attention + transition, it can be represented as ( , If it's a recurrent neural network, you need to calculate first. , then calculate and The self-attention layer of the transformer can be computed simultaneously. , Then calculate t+1, so that the output of each self-attention + transition is... It can be represented as:

[0017]

[0018]

[0019] Here, the transition function can be either a fully connected layer, as before, or a convolutional layer.

[0020] The Universal-Transformer's position embedding takes into account both the position and time dimensions of the character, and the coordinate embedding is performed again in each loop.

[0021] Preferably, in step S33, the output data is input into the CRF model, and the probabilities of classifying various word positions for each character vector in the Chinese sentence are revised according to the context information to obtain the target model. CRF makes each element of the labeled sequence dependent on the interconnected features in the observed sequence, and the model can assign different weights to each feature based on its importance. Constraints can be added to the model to ensure the effectiveness of the final prediction result. These constraints can be automatically learned by CRF during training. Given an observed sequence X, if the conditional probability distribution p(Y|X) of the target sequence Y satisfies the Markov property:

[0022]

[0023] Then p(Y|X) is a linear chain conditional random field. In the labeling problem, the probability of a specific state sequence for a given input observable sequence is defined as:

[0024]

[0025] in, Let be the transfer function, representing the observed sequence in its labeled sequence. and The transition probability between them Let be the state function, representing the probability that the label at position i in the observation sequence X is y:. , These correspond to the weights of the respective functions and need to be estimated from the training samples.

[0026] The beneficial effects of this invention are as follows:

[0027] This paper utilizes a combination of Universal-Transformer and Conditional Random Field (CRF) methods for Chinese word segmentation. By fully leveraging the advantages of each algorithm and continuously updating the model parameters through a large training corpus, the paper leverages the powerful learning capabilities of the model to achieve more accurate Chinese word segmentation results. Attached Figure Description

[0028] Figure 1 This is a flowchart of a method for applying deep learning to Chinese word segmentation proposed in this invention;

[0029] Figure 2 This is a flowchart illustrating the preprocessing process of the training corpus data set for a method of applying deep learning to Chinese word segmentation proposed in this invention.

[0030] Figure 3This is a diagram of the Universal-Transformer network structure of a method for applying deep learning to Chinese word segmentation proposed in this invention. Detailed Implementation

[0031] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0032] Reference Figure 1-3 A method for applying deep learning to Chinese word segmentation includes the following steps:

[0033] S1: Data preprocessing: The document is segmented using existing punctuation marks, paragraph separators, and other symbols that have a segmenting function. The word features of sentences are extracted to obtain the first data.

[0034] S2: Construct a dictionary for model input, which consists of two parts: a source dictionary src_dict and a target dictionary tgt_dict. Further process the first data to construct the dictionary for model input and obtain the second data.

[0035] S3: Use the model for training. Input src_dict and tgt_dict into the training model. Use the BPEmd model to perform word embedding pre-training on the preprocessed data. The first data is used as the input of Universal-Transformer to obtain the feature vector after the dimension is increased. The second data is used as the input of CRF layer to obtain the target word vector model.

[0036] S1 character features include four word positions: the first character (B), the middle character (M), the last character (E), and single-character words (S). The target corpus data is labeled with word positions to obtain the first data. The corpus data is converted into JSON format. To accelerate training, the first data is converted into HDF5 format to segment the document using delimiters. The training corpus is first segmented into short sentence sequences, and then the word position of each character is extracted to obtain character vectors. This can effectively reduce the amount of manual feature engineering in Chinese word segmentation and improve the accuracy of word segmentation.

[0037] For example, if the unprocessed text format is: Programmer\n Zhang Moumou\nr \w, it can be converted to: Programmer Zhang Moumou using the above data. BN IN IN B-NR I-NR I-NR SW.

[0038] In S2, src_dict is the single character counted based on the threshold after traversing the first data, and tgt_dict is the word position corresponding to the single character counted based on the threshold. Here, the threshold is defined as 5, and words with values ​​less than this threshold will be ignored.

[0039] The process involves splitting the corpus data into sentence sequences using spaces, then further splitting these sequences into two subsequences: one containing a single character and the other containing word position tags corresponding to that character. The two subsequences are then iterated through, and the frequency of each character is counted. If the frequency is less than a threshold of 5, the character is ignored. This process yields the source dictionary `src_dict` and the target dictionary `tgt_dict`. Each dictionary details the dictionary name, the number of characters, the character splitting criteria, and specifies that English characters are uniformly converted to lowercase. For out-of-vocabulary words, a specific format is used. <unk>Uniform labeling.

[0040] S3 uses a model for training, resulting in the following trained models:

[0041] S31: Use the BPEmd model to perform character embedding pre-training on the preprocessed data, which increases the character feature dimension of the character vectors, and obtains the feature vectors with increased dimension.

[0042] S32: The first data in S1 is used as the input of Universal-Transformer. The model encodes a certain character and performs operations with other characters in the sequence to obtain the syntactic and semantic relationship encoding between the character and the characters to the left and right in the sequence, thus obtaining a vector that extracts the contextual relationship features of the character.

[0043] S33: The second data in S1 is used as the input of the CRF layer. The probability of classifying various word positions of each character vector in the corpus data is revised according to the context information to obtain the target character vector model.

[0044] Among them, the BPEmd model or BPE encoding is a simple form of data compression that replaces consecutive bytes with bytes that are not present in the data. The original data is reconstructed using a replacement table when used later. Specifically, a sufficiently large training corpus dataset is prepared, the desired vocabulary size is determined, words are split into character sequences and suffixes "" are added to the end, and word frequencies are counted. Currently, the granularity is characters. For example, if the frequency of "low" is 5, then we rewrite it as "low ": 5. The frequency of each consecutive byte pair is counted, and the highest frequency pair is selected and merged into a new one. The above steps are repeated until the set vocabulary size is reached or the frequency of the next highest frequency byte pair is 1. The stop character "" indicates a word suffix. As the number of merging increases, the size of this table usually increases first and then decreases.

[0045] In S32, the Universal-Transformer model takes a character sequence as input. The encoder and decoder extract semantic and syntactic features between the characters and output another character sequence based on the task type. The sentence input from the encoder first passes through a self-attention layer. This layer helps the encoder pay attention to other characters in the input sequence when encoding each character. The output of the self-attention layer is passed to a feed-forward neural network, which is a transition layer that is cyclically computed using a weight-sharing method. Here, "cyclic" refers to a depth-based loop; the feed-forward neural network corresponding to each word position is exactly the same. The encoder's output serves as the decoder's input. The decoder also contains the encoder's self-attention layer and feed-forward layer. Between these two layers is another attention layer used to focus on relevant parts of the input sequence. For example, a sequence (a,b,c,d) is first represented by embedding as ( , After passing through a layer of attention + transition, it can be represented as ( , If it's a recurrent neural network, you need to calculate first. , then calculate and The self-attention layer of the transformer can be computed simultaneously. , Then calculate t+1, so that the output of each self-attention + transition is... It can be represented as:

[0046]

[0047]

[0048] Here, the transition function can be either a fully connected layer, as before, or a convolutional layer.

[0049] The Universal-Transformer's position embedding takes into account both the position and time dimensions of the character, and the coordinate embedding is performed again in each loop.

[0050] In S33, the output data is input into the CRF model. The probabilities of classifying various word positions for each character vector in the Chinese sentence are revised based on contextual information to obtain the target model. CRF makes each element of the labeled sequence dependent on the interconnected features in the observed sequence, and the model can assign different weights to each feature based on its importance. Unlike the maximum entropy model, which normalizes at each node and thus can only find a local optimum, CRF performs a global normalization operation on all features, thus solving the label bias problem. Some constraints can be added to the model to ensure that the final prediction result is effective. These constraints can be automatically learned by CRF during training. Given the observed sequence X, if the conditional probability distribution p(Y|X) of the target sequence Y satisfies the Markov property:

[0051]

[0052] Then p(Y|X) is a linear chain conditional random field. In the labeling problem, the probability of a specific state sequence for a given input observable sequence is defined as:

[0053]

[0054] in, Let be the transfer function, representing the observed sequence in its labeled sequence. and The transition probability between them Let be the state function, representing the probability that the label at position i in the observation sequence X is y:. , These correspond to the weights of the respective functions and need to be estimated from the training samples.

[0055] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.< / unk>

Claims

1. A method for applying deep learning to Chinese word segmentation, characterized in that, Includes the following steps: S1: Data preprocessing: Using the existing punctuation marks and paragraph separators in the text, the document is segmented to extract the word features of sentences and obtain the first data. S2: Construct a dictionary for model input, which consists of two parts: a source dictionary src_dict and a target dictionary tgt_dict. Further process the first data to construct the dictionary for model input and obtain the second data. S3: Use the model for training. Input src_dict and tgt_dict into the training model. Use the BPEmd model to perform word embedding pre-training on the preprocessed data. The first data is used as the input of Universal-Transformer to obtain the feature vector after the dimension is increased. The second data is used as the input of CRF layer to obtain the target word vector model. In step S3, a model is used for training, and the trained model includes: S31: Use the BPEmd model to perform character embedding pre-training on the preprocessed data, which increases the character feature dimension of the character vectors, and obtains the feature vectors with increased dimension. S32: The first data in S1 is used as the input of Universal-Transformer. The model encodes a certain character and performs operations with other characters in the sequence to obtain the syntactic and semantic relationship encoding between the character and the characters to the left and right in the sequence, thus obtaining a vector that extracts the contextual relationship features of the character. S33: The second data in S2 is used as the input of the CRF layer. The probability of classifying various word positions of each character vector in the corpus data is revised according to the context information to obtain the target character vector model. In S32, the Universal-Transformer model takes a character sequence as input. After the encoder and decoder extract the semantic and syntactic features between the characters, it outputs another character sequence according to the task type. The sentence input from the encoder first passes through a self-attention layer. This layer helps the encoder pay attention to other characters in the input sequence when encoding each character. The output of the self-attention layer is passed to the feedforward neural network. The feedforward network is a transition layer that is cyclically computed using a shared weight method. Here, "cyclic" refers to a deep recursion, where the feedforward neural network corresponding to each word position is exactly the same. The output of the encoder serves as the input to the decoder. The decoder also contains the encoder's self-attention layer and feedforward layer. Between these two layers, there is another attention layer used to focus on the relevant parts of the input sequence. For the sequence (a,b,c,d), it is first represented by embedding as ( , After passing through a layer of attention + transition, it can be represented as ( , If it's a recurrent neural network, you need to calculate first. , then calculate and The self-attention layer of the transformer can be computed simultaneously. , Then calculate t+1, so that the output of each self-attention + transition is... It can be represented as: ； ； Here, the transition function can be a fully connected layer, just like before, or it can be a convolutional layer; The Universal-Transformer's position embedding considers both the position and time dimensions of the character, and the coordinate embedding is performed again in each loop. In step S33, the output data is input into the CRF model. The probabilities of classifying various word positions for each character vector in the Chinese sentence are revised based on contextual information to obtain the target model. CRF makes each element of the labeled sequence dependent on the interconnected features in the observed sequence, and the model can assign different weights to each feature based on its importance. Constraints can be added to the model to ensure the effectiveness of the final prediction result. These constraints can be automatically learned by CRF during training. Given an observed sequence X, if the conditional probability distribution p(Y|X) of the target sequence Y satisfies the Markov property: ； Let p(Y|X) be a linear chain conditional random field. In the labeling problem, the probability of a specific state sequence for an input observation sequence is defined as: ； in, Let be the transfer function, representing the observed sequence in its labeled sequence. and The transition probability between them Let be the state function, representing the probability that the label at position i in the observation sequence X is y:. , These correspond to the weights of the respective functions and need to be estimated from the training samples.

2. The method for applying deep learning to Chinese word segmentation according to claim 1, characterized in that, The character features in S1 include four word positions: the first character of the word (B), the middle character of the word (M), the last character of the word (E), and the single character forming the word (S). The target corpus data is labeled by word position to obtain the first data. The corpus data is converted into JSON format. In order to accelerate training, the first data is converted into HDF5 format.

3. The method for applying deep learning to Chinese word segmentation according to claim 2, characterized in that, In step S2, the corpus data set is divided into sentence sequences using spaces, and then the sentence sequences are divided into two sequences using tabs "\t": one for a single character and the other for a word position label corresponding to the character. The two sequences are traversed, and the frequency of each character is counted. If the frequency is less than the threshold of 5, it is ignored. Finally, the source dictionary src_dict and the target dictionary tgt_dict are obtained.

4. The method for applying deep learning to Chinese word segmentation according to claim 1, characterized in that, Prepare a sufficiently large training corpus dataset, determine the desired vocabulary size, split words into character sequences and add a suffix "" to the end, count word frequencies, count the frequency of each consecutive byte pair, select the highest frequency pair and merge them into a new one, repeat the above steps until the set vocabulary size is reached or the frequency of the next highest frequency byte pair is 1. The stop character "" represents a word suffix, and the frequency increases with the number of merging operations.

Images