A Chinese word segmentation method, device and storage medium

Abstract

The application discloses a Chinese word segmentation method, device and storage medium, and belongs to the technical field of natural language processing. The Chinese word segmentation method comprises the following steps: S1, a second language translation sentence of a to-be-detected sentence is acquired; S2, a Chinese Bert pre-training language model is used to code the to-be-detected sentence, so as to acquire vector representation of semantic information of the whole sentence and a sentence vector representation sequence; S3, a second language Bert pre-training language model is used to code the translation sentence, so as to acquire vector representation of semantic information of the whole sentence; S4, semantic features of the to-be-detected sentence and the translation sentence are fused, so as to acquire a predicted category of each word of the to-be-detected sentence; and S5, the to-be-detected sentence is segmented according to the predicted category, so as to acquire a word segmentation result. The method improves the accuracy of word segmentation, and has a good word segmentation effect on foreign words in particular.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically to a method, apparatus, and storage medium for Chinese word segmentation. Background Technology

[0002] When processing text objects, a crucial issue lies in the "word segmentation" step, as almost all subsequent results depend on this initial step. Therefore, the accuracy of word segmentation significantly impacts subsequent processing, and different segmentation results affect feature extraction. Currently, many word segmentation modules can handle most general-purpose corpora; however, they still struggle with two types of text sets: online documents and domain-specific documents. These two types of text are characterized by containing a large number of new words, which are generally poorly covered by standard dictionaries. Therefore, improving the accuracy of Chinese word segmentation is a problem we urgently need to address. Summary of the Invention

[0003] The purpose of this invention is to overcome the above-mentioned technical deficiencies and provide a method, apparatus and storage medium for Chinese word segmentation, thereby solving the technical problem of how to improve the accuracy of Chinese word segmentation in the prior art.

[0004] To achieve the above-mentioned technical objectives, the present invention provides a method for Chinese word segmentation, comprising the following steps:

[0005] S1. Obtain the second language translation of the sentence to be detected;

[0006] S2. Use the Chinese BERT pre-trained language model to encode the sentence to be detected, and obtain the vector representation of the semantic information of the whole sentence and the sentence vector representation sequence.

[0007] S3. Use the second language BERT pre-trained language model to encode the translated sentence and obtain the vector representation of the semantic information of the whole sentence;

[0008] S4. Integrate the semantic features of the sentence to be detected and the translated sentence to obtain the predicted category of each character in the sentence to be detected;

[0009] S5. According to the predicted category, the sentence to be detected is segmented to obtain the word segmentation results.

[0010] Further, in step S1, obtaining the translated sentence of the sentence to be detected includes:

[0011] S11. Read the sentence to be detected and obtain the search statement through translation;

[0012] S12. Based on the search statement, perform a fuzzy search in the second language text database to obtain the most similar second language translation sentence.

[0013] Furthermore, in step S4, the prediction categories include word beginning, word middle, word ending, and standalone word.

[0014] Further, in step S4, the predicted category is obtained by the following steps:

[0015] S41. The vector representation sequence of the sentence to be detected and the overall semantic vector representation of the translated sentence are concatenated to obtain the fused vector representation sequence.

[0016] S42. The fused vector representation sequence is used as the input to the encoding end and fed into the BilSTM network to obtain its hidden layer output, thereby obtaining the probability distribution matrix of the predicted label.

[0017] S43. Obtain the dependency relationship between neighboring labels through the CRF layer, generate an optimal prediction sequence, and obtain the predicted category of each word based on the output prediction label sequence.

[0018] Furthermore, in step S43, the Viterbi algorithm using dynamic programming is used to solve for the optimal prediction sequence: Where argmax represents the function that finds the optimal result, and Y* is the output predicted label sequence. This indicates that sentence X in the entire sequence is equal to a certain label. The score.

[0019] Furthermore, in step S43, before obtaining the predicted category of each character based on the output predicted label sequence, a score is also given for the sentence belonging to a certain predicted category:

[0020]

[0021] Where A is a (k+2)×(k+2) transition fraction matrix, and P is the emission matrix. Representing label y i Transform to y i+1 The score, Represents the yth position of character i i The score of each tag;

[0022] Using Softmax, we obtain the probability that the label of sentence X is equal to that of Y after normalization:

[0023]

[0024] in, This indicates that sentence X in the entire sequence is equal to a certain label. The score, Y X This represents all possible label sequences; model training is achieved by maximizing the log-likelihood function, and maximum likelihood estimation is used to solve for the maximum posterior probability P(Y|X).

[0025]

[0026] Further, in step S42, the expression for the probability distribution matrix is: Where P∈R m×n n is the sequence length, and m is equal to the number of predicted categories; and These represent the output vectors in the forward and backward directions of the BiLSTM, respectively.

[0027] Furthermore, in step S2, the Bert pre-trained language model encodes the text sequence on a character-by-character basis.

[0028] Furthermore, this invention also proposes a Chinese word segmentation apparatus, comprising:

[0029] The acquisition unit is used to acquire the second language translation of the sentence to be detected;

[0030] The first encoding unit is used to encode the sentence to be detected using a Chinese BERT pre-trained language model to obtain the vector representation of the semantic information of the entire sentence and the sentence vector representation sequence.

[0031] The second encoding unit is used to encode the translated sentence using a second language BERT pre-trained language model to obtain a vector representation of the semantic information of the entire sentence.

[0032] The fusion unit is used to fuse the semantic features of the sentence to be detected and the translated sentence to obtain the predicted category of each character in the sentence to be detected;

[0033] The segmentation unit is used to segment the sentence to be detected according to the predicted category to obtain the word segmentation results.

[0034] Furthermore, the present invention also proposes a storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the above-described Chinese word segmentation method.

[0035] Compared with the prior art, the beneficial effects of the present invention include: The present invention first uses the prior knowledge of BERT to extract the vector representations of the text to be detected and the translation, and fuses the feature information of the two dimensions as the final semantic representation of the text to obtain the predicted category of each character in the sentence to be detected. According to the predicted category, the sentence to be detected is segmented to obtain the word segmentation result, which improves the accuracy of word segmentation, especially for foreign words. Attached Figure Description

[0036] Figure 1 This is a flowchart of a Chinese word segmentation method proposed in a specific embodiment of the present invention.

[0037] Figure 2 This is a schematic diagram of the structure of a Chinese word segmentation device proposed in a specific embodiment of the present invention. Detailed Implementation

[0038] Combination Figure 1 This specific embodiment provides a Chinese word segmentation method, including the following steps:

[0039] S1. Obtain the second language translation of the sentence to be detected; the obtained translation includes:

[0040] S11. Read the sentence to be detected and obtain the search statement through translation;

[0041] S12. Based on the search statement, perform a fuzzy search in the second language text database to obtain the most similar second language translation sentence;

[0042] S2. Use the Chinese BERT pre-trained language model to encode the sentence to be detected in units of characters, and obtain the vector representation of the semantic information of the whole sentence and the sentence vector representation sequence.

[0043] S3. Use the second language BERT pre-trained language model to encode the translated sentence and obtain the vector representation of the semantic information of the whole sentence;

[0044] S4. Merge the semantic features of the sentence to be detected and the translated sentence to obtain the predicted category of each character in the sentence to be detected; the predicted category includes word beginning, word middle, word ending, and standalone word; the predicted category is obtained by the following steps:

[0045] S41. The vector representation sequence of the sentence to be detected and the overall semantic vector representation of the translated sentence are concatenated to obtain the fused vector representation sequence.

[0046] S42. The fused vector representation sequence is used as the input to the encoding end and fed into the BilSTM network to obtain its hidden layer output, thereby obtaining the probability distribution matrix of the predicted label.

[0047] S43. Obtain the dependencies between neighboring labels through the CRF layer, generate an optimal prediction sequence, and obtain the predicted category of each word based on the output predicted label sequence; use the Viterbi algorithm with dynamic programming to solve for the optimal prediction sequence: Where argmax represents the function that finds the optimal result, and Y* is the output predicted label sequence. This indicates that sentence X in the entire sequence is equal to a certain label. The score;

[0048] Before obtaining the predicted category for each character based on the output predicted label sequence, the process also includes scoring the sentence to belong to a certain predicted category:

[0049]

[0050] Where A is a (k+2)×(k+2) transition fraction matrix, and P is the emission matrix. Representing label y i Transform to y i+1 The score, Represents the yth position of character i i The score of each tag;

[0051] Using Softmax, we obtain the probability that the label of sentence X is equal to that of Y after normalization:

[0052]

[0053] in, This indicates that sentence X in the entire sequence is equal to a certain label. The score, Y X This represents all possible label sequences; model training is achieved by maximizing the log-likelihood function, and maximum likelihood estimation is used to solve for the maximum posterior probability P(Y|X).

[0054]

[0055] Further, in step S42, the expression for the probability distribution matrix is: Where P∈R m×n n is the sequence length, and m is equal to the number of predicted categories; and These represent the output vectors in the forward and backward directions of the BiLSTM, respectively.

[0056] S5. According to the predicted category, the sentence to be detected is segmented to obtain the word segmentation results.

[0057] Combination Figure 2 This specific embodiment of a Chinese word segmentation device includes:

[0058] The acquisition unit is used to acquire the second language translation of the sentence to be detected;

[0059] The first encoding unit is used to encode the sentence to be detected using a Chinese BERT pre-trained language model to obtain the vector representation of the semantic information of the entire sentence and the sentence vector representation sequence.

[0060] The second encoding unit is used to encode the translated sentence using a second language BERT pre-trained language model to obtain a vector representation of the semantic information of the entire sentence.

[0061] The fusion unit is used to fuse the semantic features of the sentence to be detected and the translated sentence to obtain the predicted category of each character in the sentence to be detected;

[0062] The segmentation unit is used to segment the sentence to be detected according to the predicted category to obtain the word segmentation results.

[0063] This specific embodiment also proposes a storage medium on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described Chinese word segmentation method.

[0064] This invention provides a Chinese word segmentation method for online and domain-specific documents. It utilizes the translated information of the document to be analyzed and employs the Bert-CRF model to improve segmentation accuracy. It is particularly effective for foreign words.

[0065] The input vector of the BERT model consists of the sum of token vectors, segment vectors, and position vectors. For any input text sequence, BERT encodes it word by word, adding a [CLS] symbol at the beginning of the sequence. The [CLS] vector represents the information of the entire sentence. A [SEP] symbol is added at the end of each sentence to separate two sentences. The position vector encodes the position information of the token in the sequence as a feature vector. Unlike the Transformer method, which uses sine and cosine functions of different frequencies to generate position vectors, the BERT model uses random initialization to represent position information and optimizes it together with the model. The segment vector distinguishes the sentence to which a token belongs. Each token in the previous sentence is represented by 0, and each token in the next sentence is represented by 1. If there is only one sentence, each token is represented by 0. The token vector maps each token in the dictionary to a fixed-dimensional vector. Similar to the position vector, it is also randomly initialized and optimized together with the model. BERT captures latent semantic information in text by using sentence context to predict the mask token or the contextual relationship between two sentences in the input, through pre-training on a large corpus. This information is context-dependent and semantically rich.

[0066] To make the objectives, technical solutions, and advantages of this invention clearer, the method of this invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0067] Example 1

[0068] This embodiment proposes a method for Chinese word segmentation, including the following steps:

[0069] Step S1: Obtain the translation of the sentence to be tested;

[0070] Read the sentence to be detected and translate it to obtain the search statement;

[0071] The most similar second-language translation sentence is obtained by fuzzy searching the second-language text database based on the search query.

[0072] Step S2: Encode the sentence to be detected using a Chinese BERT pre-trained language model to obtain the vector representation of the semantic information of the entire sentence HA[cls] and the sentence vector representation sequence HA[bert] = (h1, h2, ..., hi..., hm);

[0073] The BERT model encodes text sequences character by character, and the [CLS] symbol added at the beginning of the sequence is used to represent the semantic information of the entire sentence in downstream tasks. For a sentence to be detected containing m characters, after being mapped by the BERT layer, a vector representation HA[cls] containing the semantic information of the entire sentence and a vector representation sequence of the sentence text HA[bert] = (h1, h2, ..., hi..., hm) are obtained, where hi represents the vector representation of the i-th character.

[0074] Step S3: Encode the translated sentence using a second language BERT pre-trained language model to obtain the vector representation HB[cls] of the semantic information of the entire sentence.

[0075] Step S4: Fuse the semantic features of the sentence to be detected and the translated sentence to obtain the predicted category Y = (y1, y2, ..., yn) for each character in the sentence to be detected.

[0076] 1) The vector representation sequence HA[bert] of the sentence to be detected and the overall semantic vector representation HB[cls] of the translated sentence are concatenated to obtain the fused vector representation sequence H.

[0077]

[0078] 2) The fused vector representation sequence H is used as the input to the encoder and fed into the BilSTM network to obtain its hidden layer output:

[0079]

[0080] and Let P represent the output vectors in the forward and backward directions of the BiLSTM, respectively. Then, the two vectors are concatenated and multiplied by a weight matrix W0 to obtain the probability distribution matrix P of the predicted label.

[0081]

[0082] Where P∈R m×n, where n is the sequence length and k is equal to the number of predicted categories.

[0083] 3) Obtain the dependencies between neighboring labels through the CRF layer to generate an optimal prediction sequence. Let y = (y1, y2, ..., yn) be a label sequence of length equal to the sentence length. Then, the model's score for a sentence x with label y equal to y is:

[0084]

[0085] Where (k+2)×(k+2) is the transition fraction matrix, and P is the emission matrix. Representing label y i Transform to y i+1 The score, Represents the yth position of character i i The score of each label.

[0086] Using Softmax, we obtain the probability that the label of sentence X is equal to that of Y after normalization:

[0087]

[0088] in, This indicates that sentence X in the entire sequence is equal to a certain label. The score, Y X Represents all possible label sequences;

[0089] Model training is achieved by maximizing the log-likelihood function, and maximum likelihood estimation is used to solve for the maximum posterior probability P(Y|X).

[0090]

[0091] The model uses the Viterbi algorithm of dynamic programming to solve for the optimal path during the prediction process (decoding):

[0092]

[0093] Here, argmax represents the function that finds the optimal result, and y* is the output predicted label sequence.

[0094] Each character is predicted to fall into one of the following categories: B (beginning of a word), M (middle of a word), E (end of a word), or S (standing alone as a word).

[0095] Step S5: Segment the sentence to be detected according to the predicted category to obtain the word segmentation results.

[0096] This invention first utilizes the prior knowledge of BERT to extract the hidden layer vector representations of the text to be detected and the translation. The feature information of the two dimensions is fused as the final semantic representation of the text and fed into the BilSTM-CRF model to obtain word segmentation, which improves the accuracy of word segmentation, especially for foreign words.

[0097] The specific embodiments of the present invention described above do not constitute a limitation on the scope of protection of the present invention. Any other corresponding changes and modifications made in accordance with the technical concept of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for Chinese word segmentation, characterized in that, Includes the following steps: S1. Obtain the second language translation of the sentence to be detected; S2. Use the Chinese BERT pre-trained language model to encode the sentence to be detected, and obtain the vector representation of the semantic information of the whole sentence and the sentence vector representation sequence. S3. Use the second language BERT pre-trained language model to encode the translated sentence and obtain the vector representation of the semantic information of the whole sentence; S4. Integrate the semantic features of the sentence to be detected and the translated sentence to obtain the predicted category of each character in the sentence to be detected; In step S4, the predicted category is obtained by the following steps: S41. The vector representation sequence of the sentence to be detected and the overall semantic vector representation of the translated sentence are concatenated to obtain the fused vector representation sequence. S42. The fused vector representation sequence is used as the input to the encoding end and fed into the BilSTM network to obtain its hidden layer output, thereby obtaining the probability distribution matrix of the predicted label. S43. Obtain the dependency relationship between neighboring labels through the CRF layer, generate an optimal prediction sequence, and obtain the predicted category of each character based on the output prediction label sequence; In step S43, before obtaining the predicted category of each character based on the output predicted label sequence, a score is also given for the sentence belonging to a certain predicted category: Where A is a (k+2)×(k+2) transition fraction matrix, and P is the emission matrix. Representing label y i Transform to y i+1 The score, Represents the yth position of character i i The score of each tag; Using Softmax, we obtain the probability that the label of sentence X is equal to that of Y after normalization: in, This indicates that sentence X in the entire sequence is equal to a certain label. The score, Y X This represents all possible label sequences; model training is achieved by maximizing the log-likelihood function, and maximum likelihood estimation is used to solve for the maximum posterior probability P(Y|X). S5. According to the predicted category, the sentence to be detected is segmented to obtain the word segmentation results.

2. The Chinese word segmentation method according to claim 1, characterized in that, In step S1, obtaining the translated sentence of the sentence to be detected includes: S11. Read the sentence to be detected and obtain the search statement through translation; S12. Based on the search statement, perform a fuzzy search in the second language text database to obtain the most similar second language translation sentence.

3. The Chinese word segmentation method according to claim 1, characterized in that, In step S4, the prediction categories include word beginning, word middle, word ending, and standalone word.

4. The Chinese word segmentation method according to claim 1, characterized in that, In step S43, the Viterbi algorithm using dynamic programming is used to solve for the optimal predicted sequence: Where argmax represents the function that finds the optimal result, and Y* is the output predicted label sequence. This indicates that sentence X in the entire sequence is equal to a certain label. The score.

5. The Chinese word segmentation method according to claim 1, characterized in that, In step S42, the expression for the probability distribution matrix is: Where P∈R m×n n is the sequence length, and m is equal to the number of predicted categories; and These represent the output vectors in the forward and backward directions of the BiLSTM, respectively.

6. The Chinese word segmentation method according to claim 1, characterized in that, In step S2, the Bert pre-trained language model encodes the text sequence on a character-by-character basis.

7. A Chinese word segmentation apparatus for implementing the steps of the Chinese word segmentation method as described in any one of claims 1 to 6, characterized in that, include: The acquisition unit is used to acquire the second language translation of the sentence to be detected; The first encoding unit is used to encode the sentence to be detected using a Chinese BERT pre-trained language model to obtain the vector representation of the semantic information of the entire sentence and the sentence vector representation sequence. The second encoding unit is used to encode the translated sentence using a second language BERT pre-trained language model to obtain a vector representation of the semantic information of the entire sentence. The fusion unit is used to fuse the semantic features of the sentence to be detected and the translated sentence to obtain the predicted category of each character in the sentence to be detected; The segmentation unit is used to segment the sentence to be detected according to the predicted category to obtain the word segmentation results.

8. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the Chinese word segmentation method as described in any one of claims 1 to 6.

Images