Text processing model training method, text processing method and related equipment
By combining the reference labels and point-to-point information of the training text, the text processing model is trained to identify the probabilistic relationship of word, which solves the problem of low accuracy of Chinese word participle and part-of-speech labeling, and achieves efficient Chinese word participle and new word recognition.
Patent Information
- Application Number
- CN202310619279.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-29
- Publication Date
- 2024-11-29
AI Technical Summary
There is a problem of low accuracy in Chinese word segmentation processing, especially in the absence of obvious separating symbols, it is difficult for the prior art to effectively identify words and annotate part of speech, especially the ability to recognize new words is insufficient.
By determining the mutual information between reference labels and points of the training text, the text processing model is trained to learn the probability relationship of words, and combined with the difference training of predicted labels and reference labels, efficient word part-of-word annotation of Chinese text.
It improves the accuracy of Chinese word segmentation and new word recognition ability, enhances the generalization ability of the model, can effectively process words of any length, and achieves efficient Chinese word segmentation and part-of-speech labeling.
Smart Images

Figure CN119047464A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of natural language processing technology, and specifically relates to a text processing model training method, a text processing method and related equipment. Background Art
[0002] Natural language processing requires computers to accurately extract the meaning of sentences. The processing process involves syntax, semantic components, semantic results, context, etc. The basis of all this is to accurately segment a sentence into a combination of multiple words. In the deeper natural language processing process, such as personalized recommendations, sentiment analysis, topic classification, etc., high accuracy word segmentation is required as a prerequisite.
[0003] In the Western language system, there are obvious separators between words. For English word segmentation, natural language processing only requires a simple operation to accurately divide an English sentence into several words. Unlike English, the smallest language unit used to express meaning in Chinese is the word, and there is no obvious separator between words. Therefore, how to accurately segment text, especially Chinese text, has become one of the hot issues in the field of natural language processing. Summary of the invention
[0004] The purpose of the embodiments of the present application is to provide a text processing model training method, a text processing method and related equipment, which can achieve efficient Chinese word segmentation while realizing part-of-speech tagging and have good recognition of new words.
[0005] In order to solve the above technical problems, this application is implemented as follows:
[0006] In a first aspect, an embodiment of the present application provides a method for training a text processing model, the method comprising: determining a reference label corresponding to a training text, the training text comprising a plurality of characters and a plurality of words, each word consisting of at least one character; the reference label being used to mark the position of each character in the training text in the corresponding word and the part of speech of each word; obtaining point-to-point mutual information corresponding to the training text, the point-to-point mutual information being used to represent the probability that two adjacent characters in a plurality of characters in the training text form a word; concatenating the training text and the point-to-point mutual information and inputting them into a text processing model for word segmentation and part-of-speech prediction processing to obtain a predicted label corresponding to the training text, the predicted label being used to mark the position of each character in the training text when used to form a word and the part of speech based on the word it forms; training the text processing model based on the difference between the predicted label and the reference label; wherein the trained text processing model is used to perform word segmentation and part-of-speech tagging processing on any text to be processed.
[0007] In a second aspect, an embodiment of the present application provides a text processing method, the method comprising: obtaining a text to be processed; performing word segmentation and part-of-speech prediction processing on the text to be processed through a text processing model to obtain a prediction result corresponding to the text to be processed, the prediction result being used to mark the position of each character in the text to be processed when used to form a word and the part of speech based on the word it forms; the text processing model is obtained through the training method of the text processing model as described in the first aspect; performing word segmentation and part-of-speech tagging processing on the text to be processed according to the prediction result to obtain multiple words included in the text to be processed and the part of speech of each word.
[0008] In a third aspect, an embodiment of the present application provides a training device for a text processing model, the device comprising:
[0009] A determination module, used to determine a reference label corresponding to a training text, wherein the training text includes a plurality of characters and a plurality of words, each word consisting of at least one character; the reference label is used to mark the position of each character in the training text in the corresponding word and the part of speech of each word;
[0010] The determination module is further used to obtain the point-to-point mutual information corresponding to the training text, where the point-to-point mutual information is used to represent the probability that two adjacent characters in the plurality of characters in the training text form a word;
[0011] A training module is used to concatenate the training text and the point-to-point mutual information and input them into a text processing model for word segmentation and part-of-speech prediction to obtain a prediction label corresponding to the training text, wherein the prediction label is used to mark the position of each character in the training text when it is used to form a word and the part of speech of the word it forms; the training module is also used to train the text processing model based on the difference between the prediction label and the reference label; wherein the trained text processing model is used to perform word segmentation and part-of-speech tagging on any text to be processed.
[0012] In a fourth aspect, an embodiment of the present application provides a text processing device, the device comprising:
[0013] An acquisition module is used to obtain the text to be processed;
[0014] A prediction module, used for performing word segmentation and part-of-speech prediction processing on the text to be processed through a text processing model to obtain a prediction result corresponding to the text to be processed, wherein the prediction result is used to mark the position of each character in the text to be processed when it is used to form a word and the part of speech based on the word composed of the character; the text processing model is obtained by the training method of the text processing model as described in the first aspect;
[0015] The processing module is used to perform word segmentation and part-of-speech tagging processing on the text to be processed according to the prediction result, so as to obtain multiple words included in the text to be processed and the part of speech of each word.
[0016] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect or the second aspect are implemented.
[0017] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect or the second aspect are implemented.
[0018] In an embodiment of the present application, during the training of a text processing model, first, a reference label corresponding to a training text is determined, where the training text includes multiple characters and multiple words, and each word consists of at least one character; the reference label is used to mark the position of each character in the training text in the corresponding word and the part of speech of each word; then, the point-to-point mutual information corresponding to the training text is obtained, and the point-to-point mutual information is used to represent the probability that two adjacent characters in multiple characters of the training text form a word; the training text and the point-to-point mutual information are concatenated and input into the text processing model for word segmentation and part-of-speech prediction processing to obtain a predicted label corresponding to the training text, and the predicted label is used to mark the position of each character in the training text when used to form a word and the part of speech of the word it forms; finally, the text processing model is trained based on the difference between the predicted label and the reference label. It can be seen that when training the text processing model above, not only the reference labels of the training text are used, but also the point-to-point mutual information of the training text is integrated. The point-to-point mutual information of the training text is used to reflect the probability that two adjacent characters in the training text form a word. The text processing model is trained by integrating the point-to-point mutual information of the training text and the reference labels. This allows the text processing model to learn the probability between two characters that can form a word, greatly improving the generalization ability of the model so that the accuracy of new word segmentation can be improved when the text processing model is used to segment any text in the future.
[0019] In addition, by using the part of speech of each word in the training text and the position of each character in the training text as reference labels for training the text processing model, the text processing model can not only learn the segmentation between words and have the ability to segment words, but also learn the part of speech of words in the text. The trained text processing model is used to perform word segmentation and part-of-speech tagging on any text to be processed, thereby achieving efficient Chinese word segmentation and part-of-speech tagging at the same time. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0021] Figure 1 It is a flowchart of a training method for a text processing model provided in an embodiment of the present application.
[0022] Figure 2 It is a network structure diagram of a text processing model provided in an embodiment of the present application.
[0023] Figure 3 It is a flowchart of a text processing method provided in an embodiment of the present application.
[0024] Figure 4 It is a structural diagram of a training device for a text processing model provided in an embodiment of the present application.
[0025] Figure 5 It is a structural schematic diagram of a text processing device provided in an embodiment of the present application.
[0026] Figure 6 It is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0027] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.
[0028] In order to improve the accuracy of word segmentation, the following two solutions are commonly used in related technologies:
[0029] The first is a rule-based word segmentation method, which is generally done by language experts constructing templates based on word formation principles, semantic information, and part-of-speech information, and then matching text. Although this method has high accuracy, it is highly targeted. Rules are generally field-related. If they are applied in different fields, multiple different rules need to be written, which is a lot of work, and the complexity of manually writing and maintaining rules is relatively high;
[0030] The second is a statistical word segmentation method, which uses a large amount of corpus to calculate the word frequency, word probability, left and right adjacency entropy, adjacency change number and other statistical features to identify words in the text. Although this method is more flexible and not limited by the field, it has the disadvantages of sparse data and low accuracy.
[0031] This application study found that, given the advantages of machine learning, we can use the relevant knowledge of machine learning to fully absorb the advantages of both statistics and rules. However, it has been found in practice that this method has certain limitations on the length of words and the accuracy is not high.
[0032] Furthermore, the present application has discovered that the reference labels and point-to-point mutual information of the training text can be used together to train the text processing model, so that the text processing model can learn the characteristics that a word should have. In this way, the trained text processing model has a strong generalization ability and can accurately segment the text, especially for new words that are not labeled during the training process. It also has good recognition capabilities.
[0033] Based on this, the present application provides a training method for a text processing model. Specifically, in the process of training the text processing model, first, a reference label corresponding to a training text is determined, and the training text includes multiple characters and multiple words, and each word is composed of at least one character; the reference label is used to mark the position of each character in the training text in the corresponding word and the part of speech of each word; then, the point-to-point mutual information corresponding to the training text is obtained, and the point-to-point mutual information is used to represent the probability that two adjacent characters in multiple characters of the training text form a word; the training text and the point-to-point mutual information are concatenated and input into the text processing model for word segmentation and part-of-speech prediction processing to obtain the predicted label corresponding to the training text, and the predicted label is used to mark the position of each character in the training text when it is used to form a word and the part of speech of the word it constitutes; finally, the text processing model is trained based on the difference between the predicted label and the reference label. It can be seen that when training the text processing model above, not only the reference labels of the training text are used, but also the point-to-point mutual information of the training text is integrated. The point-to-point mutual information of the training text is used to reflect the probability that two adjacent characters in the training text form a word. The text processing model is trained by integrating the point-to-point mutual information of the training text and the reference labels. This allows the text processing model to learn the probability between two characters that can form a word, greatly improving the generalization ability of the model so that the accuracy of new word segmentation can be improved when the text processing model is used to segment any text in the future.
[0034] In addition, by using the part of speech of each word in the training text and the position of each character in the training text as reference labels for training the text processing model, the text processing model can not only learn the segmentation between words and have the ability to segment words, but also learn the part of speech of words in the text. The trained text processing model is used to perform word segmentation and part-of-speech tagging on any text to be processed, thereby achieving efficient Chinese word segmentation and part-of-speech tagging at the same time.
[0035] Figure 1 A flowchart of a method for training a text processing model provided in an embodiment of the present application is shown, and the method can be executed by an electronic device, such as a terminal device or a server device. In other words, the method can be executed by software or hardware installed in a terminal device or a server device. The server device includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. Figure 1 As shown, the method may include the following steps.
[0036] S101: Determine a reference label corresponding to a training text.
[0037] The training text includes multiple characters and multiple words, each word consists of at least one character; the reference label is used to mark the position of each character in the training text and the part of speech of each word. It should be noted that the training text mentioned in this step can refer to any one of the sample texts used to train the text processing model. Unless otherwise specified, in the subsequent introduction of the embodiments of the present application, one training text is taken as an example, and other training texts in the sample text are processed in the same way as the training text and the text processing model is trained.
[0038] In one embodiment, the position of a character in its corresponding word may include the first position and other positions. In a specific implementation, the position of a character in its corresponding word may be represented by a position symbol, for example, B is used to represent the first position or the first character of a word, I represents the second character of a word (two-character word) or any character after the second character (n-character word, n≥3), and O represents punctuation.
[0039] The part of speech of each word can include common nouns, names, adjectives, verbs, etc. In the reference tag, the part of speech of each word can be represented by a part of speech label. Each part of speech corresponds to a label, as shown in Table 1 below.
[0040] In summary, in step S101, the reference label may include the position symbol of each character in the corresponding word and the label of the part of speech of each word.
[0041] Table 1 Examples of Common Part-of-Speech Tags
[0042] Label meaning Label meaning Label meaning Label meaning n Common Noun f Direction Noun s Direction Noun t Direction Noun nr Name ns Place name nt Organization Name nw Title nz Other proper names v Common Verbs vd Adverb v Noun Verb a adjective ad Adverb an adjective d adverb m Quantifiers q quantifier r pronoun p preposition c conjunction u particle xc Other function words w Punctuation
[0043] Taking the training text "The weather is nice today." as an example, through manual annotation, first segment the text to obtain the segmentation result "[‘today’, ‘weather’, ‘nice’, ‘.’]"; then, according to the common part-of-speech tags shown in Table 1, obtain the part-of-speech tagging result "[‘t’, ‘n’, ‘a’, ‘w’]". Finally, through data conversion, obtain the reference tags corresponding to the training text, such as: B-t I-t B-n I-n B-a I-a O-w, where B represents the first character of a word, I represents the second character of a word (a two-character word) or any character after the second character (an n-character word, n≥3), and O represents punctuation. The above reference tags are just an example and can also be in other forms, which are not specifically limited in this application.
[0044] S102: Obtain the pointwise mutual information corresponding to the training text.
[0045] Pointwise Mutual Information (PMI) is used to measure the correlation between two events. In this application, pointwise mutual information is used to measure the probability that two adjacent characters among multiple characters of the training text form a word. Its calculation formula is where x and y respectively represent two adjacent characters, P(x, y) represents the probability that adjacent characters x and y form a word, and P(x) and P(y) respectively represent the probabilities of the occurrence of characters x and y. Taking the two characters "zhi" and "hu" as an example, assuming that in a 500-word sample, "zhi" appears 20 times, "hu" appears 10 times, and "zhihu" appears 5 times, then the probability of the occurrence of "zhi" is 0.04, the probability of the occurrence of "hu" is 0.02, and the probability that "zhi" and "hu" appear together is 0.01. Then the pointwise mutual information between "zhi" and "hu" is 1.09.
[0046] The greater the pointwise mutual information between two adjacent characters, the greater the probability that they form a word. For example, in the aforementioned training text "The weather is nice today.", the pointwise mutual information values between each pair of adjacent characters are 0.23, 0.14, 0.12, 0.039, 0.29, and 0.095 respectively.
[0047] S103: Concatenate the training text and the pointwise mutual information and input them into the text processing model for word segmentation and part-of-speech prediction processing to obtain the prediction tags corresponding to the training text.
[0048] The prediction tags are used to mark the positions of each character in the training text when forming words and the part-of-speech based on the words they form.
[0049] Specifically, the aforementioned training text "The weather is nice today." and the vector "0.23, 0.14, 0.12, 0.039, 0.29, 0.095" composed of the mutual information between its corresponding points are concatenated as the input of the text processing model to obtain the predicted label corresponding to the training text, such as "Bt It Bn In Ba Ia Ow", where the meaning of each symbol is the same as in the aforementioned reference label.
[0050] In one embodiment, splicing the training text and the mutual information between points and inputting them into the text processing model may include: adding a start tag before the training text, and connecting the training text and the mutual information between points through a sentence separator to obtain spliced information; and inputting the spliced information into the text processing model. The start tag is represented by [CLS], and the sentence separator can be represented by [SEP]. For example, the training text is "The weather is good today", and the mutual information between points is represented by "0.23, 0.14, 0.12, 0.039, 0.29, 0.095". The spliced information obtained by splicing the training text and the mutual information between points can be represented as "[CLS] The weather is good today [SEP] 0.23, 0.14, 0.12, 0.039, 0.29, 0.095".
[0051] In one implementation, a text processing model includes: a trainer and a classifier, which concatenates training text and point-to-point mutual information and inputs them into the text processing model for word segmentation and part-of-speech prediction to obtain a predicted label corresponding to the training text, including: concatenating the training text and the point-to-point mutual information and inputting them into the trainer for encoding respectively to obtain a first encoding vector corresponding to the training text and a second encoding vector corresponding to the point-to-point mutual information; superimposing the first encoding vector and the second encoding vector and inputting them into the classifier for mapping to obtain a predicted label corresponding to the training text.
[0052] Figure 2 A network structure diagram of a text processing model provided by an embodiment of the present application is shown. The input of the text processing model can be a text sequence in a predetermined format: [CLS] training text [SEP] point-to-point mutual information, wherein [CLS] is the start symbol; [SEP] sentence separator, taking the training text "The weather is good today." as an example, the training text can be expressed as "The weather is good today.", and the point-to-point mutual information can be expressed as "0.23, 0.14, 0.12, 0.039, 0.29, 0.095", and each value of "0.23, 0.14, 0.12, 0.039, 0.29, 0.095" corresponds to the probability that two adjacent characters in the training text form a word.
[0053] Optionally, the trainer can include a BERT model or a ROBERT model. The BERT model or the ROBERT model is a pre-trained language model that uses a bidirectional encoder representation of a transformer. The pre-trained BERT representation can be fine-tuned through an additional output layer and is suitable for building models for a wide range of tasks.
[0054] After the training text and the point-to-point mutual information are concatenated, the first encoding vector vec1 corresponding to the training text and the second encoding vector vec2 corresponding to the point-to-point mutual information are obtained by a trainer such as BERT or ROBERT, and then vec1 and vec2 are concatenated or added to obtain a vector vec, which is input into the classifier classifier to obtain the predicted label corresponding to each character in the training text. The embodiment of the present application uses the point-to-point mutual information value to measure the probability of two adjacent characters forming a word, and improves the word segmentation effect of the model by superimposing the point-to-point mutual information value.
[0055] In the embodiment of the present application, the classifier is used to perform word segmentation and part-of-speech prediction processing on the training text encoded by the trainer to obtain a predicted label. Optionally, the classifier can be a decision tree classifier, a Bayesian classifier, a nearest neighbor classifier, a support vector machine classifier, a logistic regression classifier, etc., and this application does not make specific restrictions. Among them, softmax can be selected as the activation function of the classifier, and the normalized value of each type of data is calculated to obtain a predicted label.
[0056] The training method of the text processing model provided in the embodiment of the present application is not restricted by the field, is easy to expand and has good portability. At the same time, due to the generalization ability of the model, it can have good recognition of new words (words that have not appeared in the annotated data).
[0057] S104: Training a text processing model based on the difference between the predicted label and the reference label.
[0058] The difference between the predicted label and the reference label can reflect which words or parts of speech the text processing model has predicted incorrectly during the word segmentation and part-of-speech prediction process, and then avoid the next error by adjusting the model parameters of the text processing model. Specifically, training the text processing model based on the difference between the predicted label and the reference label can include: determining a loss function based on the difference between the predicted label and the reference label; and adjusting the model parameters of the text processing model in a direction to reduce the value of the loss function. For example, the model parameters of the text processing model are adjusted until the loss function is less than a preset threshold, or the number of training times reaches a preset value, to obtain a trained text processing model.
[0059] In the embodiment of the present application, because the predicted label marks the position of each character in the training text when it is used to form a word and the part of speech based on the word it forms, the trained text processing model can not only learn the segmentation between words and have the ability to segment words, but also learn the part of speech of each character based on the word it forms. In the model recognition stage, it can be used to perform word segmentation and part-of-speech tagging on any text to be processed, and the whole process does not involve the calculation of probability, entropy, etc. in related statistical methods, and the efficiency is high. In addition, since there is no limit on the length of the words in the training text, the obtained text processing model can be used to determine words of any length, and the words are only related to the part of speech. As long as the training text contains words of various parts of speech, then for any new word, its part of speech must belong to the part of speech set contained in the training text, and it can be identified with high accuracy.
[0060] In an embodiment of the present application, during the training of a text processing model, first, a reference label corresponding to a training text is determined, where the training text includes multiple characters and multiple words, and each word consists of at least one character; the reference label is used to mark the position of each character in the training text in the corresponding word and the part of speech of each word; then, the point-to-point mutual information corresponding to the training text is obtained, and the point-to-point mutual information is used to represent the probability that two adjacent characters in multiple characters of the training text form a word; the training text and the point-to-point mutual information are concatenated and input into the text processing model for word segmentation and part-of-speech prediction processing to obtain a predicted label corresponding to the training text, and the predicted label is used to mark the position of each character in the training text when used to form a word and the part of speech of the word it forms; finally, the text processing model is trained based on the difference between the predicted label and the reference label. It can be seen that when training the text processing model above, not only the reference labels of the training text are used, but also the point-to-point mutual information of the training text is integrated. The point-to-point mutual information of the training text is used to reflect the probability that two adjacent characters in the training text form a word. The text processing model is trained by integrating the point-to-point mutual information of the training text and the reference labels. This allows the text processing model to learn the probability between two characters that can form a word, greatly improving the generalization ability of the model so that the accuracy of new word segmentation can be improved when the text processing model is used to segment any text in the future.
[0061] In addition, by using the part of speech of each word in the training text and the position of each character in the training text as reference labels for training the text processing model, the text processing model can not only learn the segmentation between words and have the ability to segment words, but also learn the part of speech of words in the text. The trained text processing model is used to perform word segmentation and part-of-speech tagging on any text to be processed, thereby achieving efficient Chinese word segmentation and part-of-speech tagging at the same time.
[0062] Figure 3 The flowchart of a text processing method provided by an embodiment of the present application is shown. The method can be executed by an electronic device, such as a terminal device or a server device. In other words, the method can be executed by software or hardware installed in the terminal device or the server device. The server device includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. Figure 3 As shown, the method may include the following steps.
[0063] S301: Obtain the text to be processed.
[0064] Among them, the text to be processed is the text that needs to be segmented. The text to be processed can be the text generated in any application scenario that needs to be segmented. For example, the text to be processed can be the text in the personalized recommendation scenario. In the personalized recommendation scenario, the text to be processed can be the user's historical browsing record text collected from the application or website that needs to be personalized recommended; after the historical browsing record text is segmented, the keywords that the user is interested in can be extracted, and then news or products can be pushed to the user based on the extracted keywords. For another example, the text to be processed can also be the text in the sentiment analysis scenario. In the sentiment analysis scenario, the text to be processed refers to the interactive text communicated between the customer and the customer service. After the segmentation of such interactive text, the emotional tendency of the customer is analyzed according to the extracted segmentation, and the customers with complaint tendency are manually intervened in time to reduce the complaint rate. For another example, the text to be processed can also be the text in other scenarios that need to be segmented, such as topic classification and public opinion analysis, and the embodiments of this application are not expanded one by one.
[0065] S302: Perform word segmentation and part-of-speech prediction processing on the text to be processed through the text processing model to obtain a prediction result corresponding to the text to be processed.
[0066] The prediction results are used to mark the position of each character in the text to be processed when it is used to form a word and the part of speech of the word it forms; the text processing model is based on Figure 2 The training method embodiment of the text processing model shown is obtained. For details, please refer to the introduction of the aforementioned embodiment, which will not be repeated here.
[0067] Specifically, for example, the text to be processed is "I will play basketball tomorrow." The text processing model is used to predict the label of each character in the text to be processed. Correspondingly, it is assumed that the prediction result can be expressed as: "Br, Bt, It, Bv, Bn, In, Ow".
[0068] S303: Perform word segmentation and part-of-speech tagging on the text to be processed according to the prediction result to obtain a plurality of words included in the text to be processed and the part of speech of each word.
[0069] According to the prediction results obtained in the previous steps, this step can obtain: 1. The word segmentation result of the text to be processed is "I / tomorrow / play / basketball / ."; 2. The parts of speech corresponding to each word are "r / t / v / n / w" respectively. So far, the model has completed the word segmentation and part-of-speech tagging of the text to be processed.
[0070] In one implementation, the text to be processed can be obtained in response to a key information extraction event. After obtaining the multiple words included in the text to be processed and the part of speech of each word, the key information extraction event can be executed based on the multiple words included in the text to be processed to output the key information corresponding to the text to be processed.
[0071] In the face of a huge amount of information, it is difficult to receive a lot of information in full, so it is necessary to filter out some interesting or representative information for reception. For example, a merchant's online store sells a new type of air conditioner. After word segmentation of hundreds of thousands of reviews, key information such as "discount", "electricity leakage", "bad attitude", "slow delivery", "water leakage" and so on are extracted to change the sales strategy; another example is document topic extraction. In some long articles, keywords are extracted to summarize the document topic.
[0072] In one implementation, based on multiple words included in the text to be processed, a key information extraction event is executed to output key information corresponding to the text to be processed, including: respectively calculating the word frequency (TF value) and inverse text frequency index (IDF value) of each word in the text to be processed; calculating the importance (TF-IDF) value of each word based on the word frequency and inverse text frequency index of each word and the importance calculation formula; selecting words whose importance values meet the threshold to generate the key information corresponding to the text to be processed.
[0073] The Term Frequency-Inverse Document Frequency (TF-IDF) algorithm is a statistical calculation method used to evaluate the importance of a word to a document. The importance of a word increases in proportion to the number of times it appears in a document, but it decreases inversely with the frequency of its appearance in the corpus. Term Frequency (TF) refers to the frequency of a given word in a document, that is, the ratio of the number of times word w appears in document d, count(w,d), to the total number of words in document d, size(d). Inverse Document Frequency (IDF) is a measure of the general importance of a word. The IDF of a particular word can be obtained by dividing the total number of documents by the number of documents containing the word, and then taking the logarithm of the quotient. That is, the logarithm of the ratio of the total number of documents n to the number of documents docs(w,D) in which word w appears. Then, the product of TF and IDF is calculated to obtain the TF-IDF value of each word. A high term frequency in a particular document and a low document frequency of the term in the entire document collection can produce a high-weight TF-IDF. Therefore, TF-IDF tends to filter out common words and retain important words. Finally, key information corresponding to the text to be processed is generated by selecting words whose TF-IDF values meet the threshold.
[0074] In the embodiment of the present application, when training the text processing model, not only the reference label of the training text is used, but also the point-to-point mutual information of the training text is integrated. The point-to-point mutual information of the training text is used to reflect the probability that two adjacent characters in the training text form a word. The point-to-point mutual information of the training text and the reference label are integrated to train the text processing model, so that the text processing model can learn the probability between two characters that can form a word, which greatly improves the generalization ability of the model, so that when the text processing model is used to segment any text in the future, the accuracy of the segmentation of new words can be improved. In addition, by using the part of speech of each word in the training text and the position of each character in the training text in the word to which it belongs as the reference label of the training text processing model, the text processing model can learn the segmentation between words, have the ability of word segmentation, and learn the part of speech of the word in the text. The text processing model after training is used to perform word segmentation and part-of-speech tagging on any text to be processed, so that efficient Chinese word segmentation can be achieved while tagging the part of speech. The text processing model obtained by the above can improve the accuracy of the word segmentation of the text to be processed.
[0075] Figure 4 A structural schematic diagram of a text processing model training device provided in an embodiment of the present application is shown. Figure 4As shown, the device 400 includes: a determination module 410 and a training module 420.
[0076] A determination module 410 is used to determine a reference label corresponding to a training text, wherein the training text includes a plurality of characters and a plurality of words, each word consisting of at least one character; the reference label is used to mark the position of each character in the training text in the corresponding word and the part of speech of each word;
[0077] The determination module 410 is further used to obtain the point-to-point mutual information corresponding to the training text, where the point-to-point mutual information is used to represent the probability that two adjacent characters in the plurality of characters of the training text form a word;
[0078] The training module 420 is used to splice the training text and the point-to-point mutual information and input them into a text processing model for word segmentation and part-of-speech prediction processing to obtain a prediction label corresponding to the training text, wherein the prediction label is used to mark the position of each character in the training text when it is used to form a word and the part of speech of the word formed by it;
[0079] The training module 420 is also used to train the text processing model based on the difference between the predicted label and the reference label; wherein the trained text processing model is used to perform word segmentation and part-of-speech tagging on any text to be processed.
[0080] In one implementation, the text processing model includes a trainer and a classifier. When the training module 420 concatenates the training text and the point-to-point mutual information and inputs them into the text processing model for word segmentation and part-of-speech prediction processing to obtain a predicted label corresponding to the training text, the following steps are performed: the training text and the point-to-point mutual information are concatenated and input into the trainer for encoding processing respectively to obtain a first encoding vector corresponding to the training text and a second encoding vector corresponding to the point-to-point mutual information; the first encoding vector and the second encoding vector are superimposed and input into the classifier for mapping processing to obtain a predicted label corresponding to the training text.
[0081] In one implementation, the training module 420 performs the following steps when training the text processing model based on the difference between the predicted label and the reference label:
[0082] A loss function is determined based on the difference between the predicted label and the reference label; and a model parameter of the text processing model is adjusted in a direction to reduce the value of the loss function.
[0083] In one implementation, the training model 420 performs the following steps when concatenating the training text and the point-to-point mutual information and inputting them into the text processing model:
[0084] A start marker is added before the training text, and the training text and the point-to-point mutual information are connected via a sentence separator to obtain concatenated information; and the concatenated information is input into a text processing model.
[0085] In an embodiment of the present application, during the training of a text processing model, first, a reference label corresponding to a training text is determined, where the training text includes multiple characters and multiple words, and each word consists of at least one character; the reference label is used to mark the position of each character in the training text in the corresponding word and the part of speech of each word; then, the point-to-point mutual information corresponding to the training text is obtained, and the point-to-point mutual information is used to represent the probability that two adjacent characters in multiple characters of the training text form a word; the training text and the point-to-point mutual information are concatenated and input into the text processing model for word segmentation and part-of-speech prediction processing to obtain a predicted label corresponding to the training text, and the predicted label is used to mark the position of each character in the training text when used to form a word and the part of speech of the word it forms; finally, the text processing model is trained based on the difference between the predicted label and the reference label. It can be seen that when training the text processing model above, not only the reference labels of the training text are used, but also the point-to-point mutual information of the training text is integrated. The point-to-point mutual information of the training text is used to reflect the probability that two adjacent characters in the training text form a word. The text processing model is trained by integrating the point-to-point mutual information of the training text and the reference labels. This allows the text processing model to learn the probability between two characters that can form a word, greatly improving the generalization ability of the model so that the accuracy of new word segmentation can be improved when the text processing model is used to segment any text in the future.
[0086] In addition, by using the part of speech of each word in the training text and the position of each character in the training text as reference labels for training the text processing model, the text processing model can not only learn the segmentation between words and have the ability to segment words, but also learn the part of speech of words in the text. The trained text processing model is used to perform word segmentation and part-of-speech tagging on any text to be processed, thereby achieving efficient Chinese word segmentation and part-of-speech tagging at the same time.
[0087] Figure 5 A structural diagram of a text processing device provided in an embodiment of the present application is shown. Figure 5 As shown, the device 500 includes: an acquisition module 510, a prediction model 520 and a processing module 530.
[0088] An acquisition module 510 is used to acquire the text to be processed;
[0089] Prediction module 520, used for performing word segmentation and part-of-speech prediction processing on the text to be processed by using a text processing model, and obtaining a prediction result corresponding to the text to be processed, wherein the prediction result is used to mark the position of each character in the text to be processed when it is used to form a word and the part-of-speech of the word formed by it; the text processing model is obtained by executing any of the training methods of the aforementioned text processing models;
[0090] The processing module 530 is used to perform word segmentation and part-of-speech tagging processing on the text to be processed according to the prediction result, so as to obtain multiple words included in the text to be processed and the part of speech of each word.
[0091] In one implementation, the text to be processed is obtained in response to a key information extraction event, and the processing module 530 is further used to: execute the key information extraction event based on multiple words included in the text to be processed, and output key information corresponding to the text to be processed.
[0092] In one implementation, the processing module 530 is used to: respectively calculate the word frequency and inverse text frequency index of each word in the text to be processed; calculate the importance value of each word according to the word frequency and inverse text frequency index of each word using the importance calculation formula; select words whose importance values meet the threshold to generate key information corresponding to the text to be processed.
[0093] In the embodiment of the present application, when training the text processing model, not only the reference label of the training text is used, but also the point-to-point mutual information of the training text is integrated. The point-to-point mutual information of the training text is used to reflect the probability that two adjacent characters in the training text form a word. The point-to-point mutual information of the training text and the reference label are integrated to train the text processing model, so that the text processing model can learn the probability between two characters that can form a word, which greatly improves the generalization ability of the model, so that when the text processing model is used to segment any text in the future, the accuracy of the segmentation of new words can be improved. In addition, by using the part of speech of each word in the training text and the position of each character in the training text in the word to which it belongs as the reference label of the training text processing model, the text processing model can learn the segmentation between words, have the ability of word segmentation, and learn the part of speech of the word in the text. The text processing model after training is used to perform word segmentation and part-of-speech tagging on any text to be processed, so that efficient Chinese word segmentation can be achieved while tagging the part of speech. The text processing model obtained by the above can improve the accuracy of the word segmentation of the text to be processed.
[0094] Figure 6A schematic diagram of the hardware structure of an electronic device that implements the embodiment of the present application is shown. Referring to the figure, at the hardware level, the electronic device includes a processor, and optionally, an internal bus, a network interface, and a memory. Among them, the memory may include a memory, such as a high-speed random access memory (Random-Access Memory, RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage, etc. Of course, the electronic device may also include hardware required for other services.
[0095] The processor, the network interface and the memory may be interconnected through an internal bus, which may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one bidirectional arrow is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0096] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.
[0097] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a device for locating the target user at the logical level. The processor executes the program stored in the memory and is specifically used to execute: each step of the text processing model training method described in at least one embodiment of the above method embodiments or each step of the text processing method described in at least one embodiment of the above method embodiments.
[0098] The methods disclosed in the embodiments shown in the flowcharts of the present application can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.
[0099] The electronic device can also execute the training method of the text processing model described in at least one embodiment of the previous method embodiments or the various steps of the text processing method described in at least one embodiment of the above method embodiments, and can achieve the same technical effect as the previous method embodiments, which will not be repeated here.
[0100] Of course, in addition to software implementation methods, the electronic device of the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0101] An embodiment of the present application also proposes a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple application programs, the electronic device executes the various steps of the text processing model training method described in at least one embodiment of the above-mentioned method embodiments or the various steps of the text processing method described in at least one embodiment of the above-mentioned method embodiments.
[0102] The computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0103] Furthermore, an embodiment of the present application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the steps of at least one embodiment of the above-mentioned method embodiments are implemented.
[0104] In short, the above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
[0105] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
[0106] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0107] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0108] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
Claims
1. A training method for a text processing model, characterized in that: The method comprises: Determine a reference tag corresponding to a training text, wherein the training text includes a plurality of characters and a plurality of words, each word consisting of at least one character; the reference tag is used to mark the position of each character in the training text in the corresponding word and the part of speech of each word; Obtaining point-to-point mutual information corresponding to the training text, wherein the point-to-point mutual information is used to represent the probability that two adjacent characters in a plurality of characters of the training text form a word; The training text and the point-to-point mutual information are spliced and input into a text processing model for word segmentation and part-of-speech prediction processing to obtain a prediction label corresponding to the training text, wherein the prediction label is used to mark the position of each character in the training text when it is used to form a word and the part of speech of the word formed by it; The text processing model is trained based on the difference between the predicted label and the reference label; wherein the trained text processing model is used to perform word segmentation and part-of-speech tagging on any text to be processed.
2. The method according to claim 1, characterized in that The text processing model includes a trainer and a classifier. The training text and the point-to-point mutual information are spliced and input into the text processing model for word segmentation and part-of-speech prediction processing to obtain a prediction label corresponding to the training text, including: The training text and the point-to-point mutual information are spliced and input into the trainer for encoding respectively to obtain a first encoding vector corresponding to the training text and a second encoding vector corresponding to the point-to-point mutual information; The first encoding vector and the second encoding vector are superimposed and input into the classifier for mapping processing to obtain a predicted label corresponding to the training text.
3. The method according to claim 2, characterized in that The step of training the text processing model based on the difference between the predicted label and the reference label comprises: determining a loss function based on a difference between the predicted label and the reference label; The model parameters of the text processing model are adjusted in a direction that reduces the value of the loss function.
4. The method according to claim 1, characterized in that: The step of splicing the training text and the point-to-point mutual information and inputting them into a text processing model comprises: Adding a start marker before the training text, and connecting the training text and the point-to-point mutual information through a sentence separator to obtain concatenated information; The concatenated information is input into the text processing model.
5. A text processing method, characterized in that: include: Get the text to be processed; The text to be processed is processed by word segmentation and part-of-speech prediction through a text processing model to obtain a prediction result corresponding to the text to be processed, wherein the prediction result is used to mark the position of each character in the text to be processed when it is used to form a word and the part-of-speech of the word composed of each character; the text processing model is obtained through the training method of the text processing model according to any one of claims 1 to 4; The text to be processed is segmented and marked with parts of speech according to the prediction result to obtain a plurality of words included in the text to be processed and the part of speech of each word.
6. The method according to claim 5, characterized in that The text to be processed is obtained in response to a key information extraction event. After obtaining a plurality of words included in the text to be processed and the part of speech of each word, the method further includes: Based on the multiple words included in the text to be processed, the key information extraction event is executed to output the key information corresponding to the text to be processed.
7. The method according to claim 6, characterized in that The step of executing the key information extraction event based on the multiple words included in the text to be processed and outputting the key information corresponding to the text to be processed includes: Calculating the word frequency and inverse text frequency index of each word in the text to be processed respectively; Calculate the importance value of each word according to the word frequency and inverse text frequency index of each word using the importance calculation formula; Select words whose importance values meet a threshold value to generate key information corresponding to the text to be processed.
8. A training device for a text processing model, characterized in that: The device comprises: A determination module, used to determine a reference label corresponding to a training text, wherein the training text includes a plurality of characters and a plurality of words, each word consisting of at least one character; the reference label is used to mark the position of each character in the training text in the corresponding word and the part of speech of each word; The determination module is further used to obtain the point-to-point mutual information corresponding to the training text, where the point-to-point mutual information is used to represent the probability that two adjacent characters in the plurality of characters in the training text form a word; A training module, used for splicing the training text and the point-to-point mutual information and inputting them into a text processing model for word segmentation and part-of-speech prediction processing to obtain a prediction label corresponding to the training text, wherein the prediction label is used to mark the position of each character in the training text when it is used to form a word and the part of speech of the word formed by it; The training module is also used to train the text processing model based on the difference between the predicted label and the reference label; wherein the trained text processing model is used to perform word segmentation and part-of-speech tagging on any text to be processed.
9. A text processing device, characterized in that: include: An acquisition module is used to obtain the text to be processed; A prediction module, used to perform word segmentation and part-of-speech prediction processing on the text to be processed through a text processing model to obtain a prediction result corresponding to the text to be processed, wherein the prediction result is used to mark the position of each character in the text to be processed when it is used to form a word and the part of speech of the word composed of the character; The text processing model is obtained by the training method of the text processing model according to any one of claims 1 to 5; The processing module is used to perform word segmentation and part-of-speech tagging processing on the text to be processed according to the prediction result, so as to obtain multiple words included in the text to be processed and the part of speech of each word.
10. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method as described in any one of claims 1-4 or 5-7.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the method according to any one of claims 1-4 or 5-7 are implemented.
