Weakly supervised word segmentation method using information entropy disambiguation
By combining the information entropy disambiguation method of dictionary and pre-trained language model, the problem of ambiguity identification in Chinese word segmentation is solved, the segmentation performance and OOV word recall rate are improved, and it is suitable for word segmentation tasks under weak supervision.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- KUNMING UNIV OF SCI & TECH
- Filing Date
- 2022-07-13
- Publication Date
- 2026-05-29
AI Technical Summary
Existing technologies in Chinese word segmentation tasks, especially in domain-specific word segmentation tasks, lack accurately labeled corpora and the cost of manual labeling is high, making it difficult to effectively solve ambiguity recognition. Deep learning algorithms, after being trained in accurate general domains, perform poorly in specific domains.
By combining existing dictionaries, pre-trained language models, and information entropy, noisy labeled data is generated through forward and backward maximum matching. Multi-label classification loss and greedy strategy decoding are used to solve the problem of ambiguous identification.
It improved the F1 score of word segmentation tasks by 2.5%, significantly increased the recall rate of OOV words by 85%, and had fast decoding speed, making it suitable for word segmentation tasks under weak supervision.
Smart Images
Figure CN115204169B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a weakly supervised word segmentation method that utilizes information entropy disambiguation, and relates to the field of natural language processing technology. Background Technology
[0002] Word segmentation is the prerequisite and foundation for natural language processing tasks. Its results are applied to natural language processing tasks such as text classification, information retrieval, and machine translation. The quality of its implementation algorithm directly affects the reliability and practicality of other downstream tasks.
[0003] Ambiguity identification is one of the challenges in word segmentation. In matching-based segmentation methods, setting association rules alone cannot effectively resolve ambiguities. Statistical disambiguation methods mainly fall into two categories: 1. Calculating statistical values (e.g., information entropy, mutual information, t-test) from dictionary-based matching results and setting thresholds for these statistical values to assist in word segmentation. 2. Obtaining the optimal segmentation decision by selecting the candidate with the highest posterior probability based on a statistical language model. This method requires setting feature templates and calculating a large amount of statistical information.
[0004] Currently, mainstream deep learning-based algorithms typically employ neural networks such as RNNs and LSTMs, or pre-trained language models for encoding, focusing on capturing contextual information. Most methods are trained on large, accurate, general-domain labeled datasets and do not fully utilize useful information from Chinese dictionaries. Meanwhile, domain-specific word segmentation has progressed slowly due to a lack of accurately labeled corpora and the high cost of manual annotation. Summary of the Invention
[0005] The purpose of this invention is to provide a weakly supervised word segmentation method that utilizes information entropy disambiguation. When precisely labeled corpora are scarce, this method combines a dictionary, simple statistical concepts, and a neural network model to solve the problem of ambiguity recognition under weakly supervised word segmentation.
[0006] To achieve the above-mentioned technical objectives and effects, the present invention is implemented through the following technical solution:
[0007] A weakly supervised word segmentation method utilizing information entropy disambiguation is proposed. This method uses an existing dictionary to perform forward and reverse maximum matching on unlabeled word segmentation data, merging the two matching results to obtain noisy labeled word segmentation data. A pre-trained language model is used to obtain the vector sequence of the input text, which is then fed into a scoring model based on the first and last characters of candidate words. The segmentation loss is calculated using multi-label classification loss, and for ambiguous parts, the loss is calculated based on minimizing information entropy. Finally, a greedy strategy is used for decoding.
[0008] A weakly supervised word segmentation method utilizing information entropy disambiguation includes the following steps:
[0009] S1: Using the existing dictionary, perform forward and reverse maximum matching on the unlabeled word segmentation data respectively, and merge the two matching results to obtain a text sequence of labeled word segmentation data with noise.
[0010] S2: Encode the text sequence t of length n using a pre-trained language model to obtain the vector sequence [e1, e2, ..., e...]. n ], through linear transformation:
[0011] h i =W h e i +b h
[0012] r i =W r e i +b r
[0013] We obtain two vector sequences h = [h1, h2, ..., ht] representing the text sequence t. n ], h∈R n×d and r = [r1, r2, ..., r n ], r∈R n×d , is the vector sequence used to identify whether a word segmentation result is formed;
[0014] The i-th vector in the word embedding matrix h represents when t i As the starting character of a continuous segment, its encoding is h. i The i-th vector in the word embedding matrix r represents the vector when t i As the end character of a continuous segment, it is encoded as r. i ;
[0015] W h and W r It is the weight matrix, b h and b r It is the bias matrix;
[0016] S3: Definition As from t i to t j continuous segments [t] i :t j The scoring function for u; i and v i It is h i and r i After performing Sinusoidal position encoding, a vector with position encoding information is obtained;
[0017] The scoring function will produce two types of results:
[0018] s(i,j)>0 indicates that this segment is a candidate segment, and s(i,j)<0 indicates that this segment cannot be a segment.
[0019] S4: Decode the score matrix obtained in S3 starting from the current i-th row:
[0020] a. When there are one or more s(i,j)>0, j≥i in the i-th row, select the position (i,j) with the largest score in the i-th row, indicating that the continuous segment with (i,j) as the index is a word;
[0021] b. Otherwise, take j = i, that is, take the continuous segment with (i, i) as the subscript as a single word;
[0022] c. Confirm that the starting character index of the next word is j+1, i.e. i←j+1, and repeat step a;
[0023] S5: To ensure that the scores s(i,j) of positions corresponding to segments that may form words in the score matrix are greater than 0, and the scores s(i,j) of positions corresponding to segments that cannot form words are less than 0, guided by the matching results obtained in S1, the loss is calculated using a multi-label classification loss function on the entire score matrix. Where P represents the set of continuous segments with scores s(i,j) > 0, and N represents the set of continuous segments with scores s(i,j) < 0.
[0024]
[0025] Since each row of the score matrix may contain multiple s(i,j) > 0 during model training, indicating the existence of multiple words starting with the character represented by the index of that row, the loss function... The fractions S = [s1, s2, ..., s] corresponding to all possible consecutive segments m Calculate the information entropy;
[0026] p = softmax(S)
[0027]
[0028] S6: Employ the Adam optimizer with an adaptive learning rate and set the parameter weight_decay to 0.00001 to adjust the impact of model complexity on the loss function and prevent overfitting.
[0029] Furthermore, S1 performs forward and backward maximum matching on the input text sequence simultaneously, takes the union of the matching results, and obtains a set of all rule-based meaningful word segmentation results, denoted as Spans = [(s f1 e f1 ), (s f2 e f2), (s b2 e b2 ), …, (s fm e fm )], where (s fi e fi ) represent the indices of the first and last characters of the i-th word in the forward matching, respectively. b2 e b2 ) represent the indices of the first and last characters of the i-th word in the reverse matching.
[0030] Furthermore, the scoring function defined in S3 uses Sinusoidal positional encoding during the scoring phase, where u k，2i v k，2i+1 These are the 2i and 2i+1th components of the encoding vector at position k, respectively, and d is the dimension of the position vector;
[0031]
[0032] The first and last character vectors u, which incorporate position encoding information i and v i Multiplying them together yields the corresponding continuous segment [t]. i :t j The score.
[0033] Furthermore, S4 must satisfy two constraints during the word segmentation process: first, in continuous segments [t] i :t j In the first case, j ≥ i; secondly, when the ending index of the previous word is m, the starting index of the next word must be m+1.
[0034] Since the S3 score matrix cannot fully satisfy the second constraint, a greedy algorithm is used for decoding in the decoding stage.
[0035] The beneficial effects of this invention are:
[0036] The weakly supervised word segmentation method of the present invention utilizes information entropy disambiguation to transform the word segmentation task into a multi-label classification task, rather than the current mainstream sequence labeling task.
[0037] The weakly supervised word segmentation method of the present invention, which utilizes information entropy disambiguation, does not directly calculate the left and right information entropy or mutual information of candidate words based on word frequency statistics to determine the final word segmentation result for possible ambiguity. Instead, it uses a scoring model to calculate the score of all candidate words, then calculates the Softmax of multiple scores, and further calculates the information entropy of the ambiguous segment, so that the difference between the score of the correct result and the scores of the others is as large as possible.
[0038] The weakly supervised word segmentation method of the present invention, which utilizes information entropy disambiguation, improves the evaluation index F1 by 2.5% and the recall rate of out-of-vocabulary (OOV) words by more than 85% compared with the baseline model.
[0039] The weakly supervised word segmentation method of the present invention utilizes information entropy disambiguation. Under the two constraints proposed, the final output score matrix of the model is decoded based on a greedy strategy, resulting in fast decoding speed.
[0040] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description
[0041] Figure 1 This is a flowchart of the weakly supervised word segmentation method using information entropy disambiguation as described in an embodiment of the present invention; Detailed Implementation
[0042] To more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below with reference to the accompanying drawings.
[0043] A weakly supervised word segmentation method utilizing information entropy disambiguation is proposed. This method uses an existing dictionary to perform forward and reverse maximum matching on unlabeled word segmentation data, merging the two matching results to obtain noisy labeled word segmentation data. A pre-trained language model is used to obtain the vector sequence of the input text, which is then fed into a scoring model based on the first and last characters of candidate words. The segmentation loss is calculated using multi-label classification loss, and for ambiguous parts, the loss is calculated based on minimizing information entropy. Finally, a greedy strategy is used for decoding.
[0044] The present invention will now be described in conjunction with specific embodiments:
[0045] Example 1
[0046] like Figure 1 As shown
[0047] A weakly supervised word segmentation method utilizing information entropy disambiguation includes the following steps:
[0048] S1: Using the existing dictionary, perform forward and reverse maximum matching on the unlabeled word segmentation data respectively, and merge the two matching results to obtain a text sequence of labeled word segmentation data with noise.
[0049] Perform forward and backward maximum matching on the input text sequence simultaneously, and take the union of the matching results to obtain the set of all rule-based meaningful word segmentation results, denoted as Spans = [(s f1 e f1 ), (s f2 e f2 ), (sb2 e b2 ), ..., (s fm e fm )], where (s fi e fi ) represent the indices of the first and last characters of the i-th word in the forward matching, respectively. b2 e b2 ) represent the indices of the first and last characters of the i-th word in the reverse matching.
[0050] S2: Encode the text sequence t of length n using a pre-trained language model to obtain the vector sequence [e1, e2, ..., e...]. n ], through linear transformation:
[0051] h i =W h e i +b h
[0052] r i =W r e i +b r
[0053] We obtain two vector sequences h = [h1, h2, ..., ht] representing the text sequence t. n ], h∈R n×d and r = [r1, r2, ..., r n ], r∈R n×d , is the vector sequence used to identify whether a word segmentation result is formed;
[0054] The i-th vector in the word embedding matrix h represents when t i As the starting character of a continuous segment, its encoding is h. i The i-th vector in the word embedding matrix r represents the vector when t i As the end character of a continuous segment, it is encoded as r. i ;
[0055] W h and W r It is the weight matrix, b h and b r It is the bias matrix;
[0056] S3: Definition As from t i to t j continuous segments [t] i :t j The scoring function for u; i and v i It is h i and r iAfter performing Sinusoidal position encoding, a vector with position encoding information is obtained;
[0057] The scoring function will produce two types of results:
[0058] s(i,j)>0 indicates that this segment is a candidate segment, and s(i,j)<0 indicates that this segment cannot be a segment.
[0059] The defined scoring function uses Sinusoidal positional encoding during the scoring phase, where u k，2i v k，2i+1 These are the 2i and 2i+1th components of the encoding vector at position k, respectively, and d is the dimension of the position vector;
[0060]
[0061] The first and last character vectors u, which incorporate position encoding information i and v i Multiplying them together yields the corresponding continuous segment [t]. i :t j The score;
[0062] S4: Decode the score matrix obtained in S3 starting from the current i-th row:
[0063] a. When there are one or more s(i,j)>0, j≥i in the i-th row, select the position (i,j) with the largest score in the i-th row, indicating that the continuous segment with (i,j) as the index is a word;
[0064] b. Otherwise, take j = i, that is, take the continuous segment with (i, i) as the subscript as a single word;
[0065] c. Confirm that the starting character index of the next word is j+1, i.e. i←j+1, and repeat step a;
[0066] S4 must satisfy two constraints during word segmentation: First, in continuous segments [t]... i :t j In the first case, j ≥ i; secondly, when the ending index of the previous word is m, the starting index of the next word must be m+1.
[0067] Since the S3 score matrix does not fully satisfy the second constraint, a greedy algorithm is used for decoding.
[0068] S5: To ensure that the scores s(i,j) of positions corresponding to segments that may form words in the score matrix are greater than 0, and the scores s(i,j) of positions corresponding to segments that cannot form words are less than 0, guided by the matching results obtained in S1, the loss is calculated using a multi-label classification loss function on the entire score matrix. Where P represents the set of continuous segments with scores s(i,j) > 0, and N represents the set of continuous segments with scores s(i,j) < 0.
[0069]
[0070] Since each row of the score matrix may contain multiple s(i,j) > 0 during model training, indicating the existence of multiple words starting with the character represented by the index of that row, the loss function... The fractions S = [s1, s2, ..., s] corresponding to all possible consecutive segments m Calculate the information entropy;
[0071] p = softmax(S)
[0072]
[0073] S6: Employ the Adam optimizer with an adaptive learning rate and set the parameter weight_decay to 0.00001 to adjust the impact of model complexity on the loss function and prevent overfitting.
[0074] Example 2
[0075] To compare with existing methods, this invention uses the Simplified Chinese word segmentation corpora PKU and MSR from the SIGHAN2005 corpus and sets up two sets of comparative experiments to demonstrate the improvement of word segmentation performance by the neural network model and the loss function for ambiguous segments, respectively. Improvements to the model's word segmentation performance. The evaluation metrics are consistent with the comparison methods, using Precision, Recall, and F1 as indicators to evaluate the word segmentation results. Statistical information for the dataset used is shown in Table 1.
[0076] Table 1 Statistics of PKU and MSR datasets
[0077]
[0078] (1) Verify the improvement of word segmentation performance by the neural network model
[0079] In this comparative experiment, the control group only used the bidirectional maximum matching algorithm. Since the number of single-character words in Chinese is much smaller than that of non-single-character words, the solution for ambiguous parts is to simultaneously perform forward and backward longest matching on the input text sequence. If the number of words in the two methods is different, the one with fewer matching attempts is returned. Otherwise, the one with fewer single characters is returned. If the number of single characters is also the same, the result of backward maximum matching is returned first. The experimental group uses the model MCLCWS proposed in this invention, and the model parameters are shown in Table 2. The experimental results are shown in Table 3.
[0080] Table 2 Baseline Model Parameters
[0081]
[0082] Table 3 shows that the experimental group's MCLCWS model improved the F1 score by 0.013 compared to the baseline model. The OOV rate was 2.6% on the MSR dataset and 5.8% on the PKU dataset. The MCLCWS model significantly improved its ability to recognize OOVs, achieving a ROOV score of 0.872 on the MSR dataset. In the baseline model, the dictionary constructed from the training set was used, and all correctly labeled OOVs in the test set could only be incorrectly labeled results due to ambiguity handling errors, resulting in extremely low recall for OOVs, while the recall for IVs was very high.
[0083] (2) Verify the loss function for ambiguous segments. Improvement of model word segmentation performance
[0084] In this comparative experiment, the control group used the MCLCWS model, while the experimental group used the MCLCWS model with a loss function. As shown in Table 3, the experimental group showed significant improvements in all evaluation metrics, including P, R, F1, and RIV. We believe... By optimizing the information entropy of candidate words that start with the same character, the score gap between the correct word segmentation and the other candidate words is widened, and the word segmentation performance of the model is significantly improved.
[0085] Table 3 Experimental Results
[0086]
[0087]
[0088] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0089] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.
Claims
1. A weakly supervised word segmentation method utilizing information entropy disambiguation, characterized in that, Includes the following steps: S1: Using the existing dictionary, perform forward and reverse maximum matching on the unlabeled word segmentation data respectively, and merge the two matching results to obtain a text sequence of labeled word segmentation data with noise. S2: Transform the length of... text sequence The vector sequence is obtained after encoding by the pre-trained language model. Through linear transformation: Obtain the text sequence Two sets of vector sequences , and , , is the vector sequence used to identify whether a word segmentation result is formed; In word embedding matrix The first in Each vector represents when As the starting character of a continuous segment, its encoding is: In the word embedding matrix The first in Each vector represents when As the end character of a continuous segment, its encoding is: ; and It is a weight matrix. and It is the bias matrix; S3: Definition As from arrive continuous segments The scoring function; and yes and After performing Sinusoidal position encoding, a vector with position encoding information is obtained; The scoring function will produce two types of results: This indicates that this segment is a candidate word segment. This means that this segment cannot form a word segment; S4: Transfer the score matrix obtained in S3 from the current... Line decoding begins: a. No. There are one or more rows , When, select the first The position with the highest row score , indicating that The consecutive segments indicated by subscripts constitute a single word; b. Otherwise, take ,Right now A continuous segment marked with a subscript is used as a single word; c. Confirm the starting character index of the next word. ,Right now Repeat step a; S5: To ensure the scores of the positions corresponding to the segments that may constitute words in the scoring matrix. The score for the position corresponding to the fragment that cannot form a word. Guided by the matching results obtained in S1, the loss is calculated using a multi-label classification loss function on the entire score matrix. ,in, Indicates score A set of consecutive segments Indicates score A set of consecutive segments: Since there may be multiple [various parameters] in each row of the score matrix during model training. This indicates the existence of multiple words that begin with the character represented by the index of that line. The loss function... The score corresponding to all possible consecutive segments Calculate the information entropy; S6: Employ the Adam optimizer with an adaptive learning rate and set the parameter weight_decay to 0.00001 to adjust the impact of model complexity on the loss function and prevent overfitting.
2. The weakly supervised word segmentation method using information entropy disambiguation as described in claim 1, characterized in that: S1 performs forward and backward maximum matching on the input text sequence simultaneously, and takes the union of the matching results to obtain a set of all rule-based meaningful word segmentation results, denoted as: ,in These represent the first and second digits in the forward matching, respectively. The indices of the first and last characters of each word. These represent the first and second parts of the reverse matching process. The indices of the first and last characters of each word.
3. The weakly supervised word segmentation method using information entropy disambiguation as described in claim 1, characterized in that: The scoring function defined in S3 uses Sinusoidal positional encoding during the scoring phase. , They are the locations The first encoding vector One portion, It is the dimension of the position vector; The first and last character vectors incorporating position encoding information and Multiply to obtain the corresponding continuous segments The score.
4. The weakly supervised word segmentation method using information entropy disambiguation as described in claim 1, characterized in that: The S4 method must satisfy two constraints during word segmentation: first, in continuous segments... middle Secondly, the ending index of the previous word is... When, the starting index of the next word must be 0. ; Since the S3 score matrix cannot fully satisfy the second constraint, a greedy algorithm is used for decoding in the decoding stage.