Model training method, text processing method and related device

By combining the reference labels and inter-point mutual information of the training text, the BERT or ROBERT model is used for Chinese word segmentation and part-of-speech prediction, which solves the problem of low word segmentation accuracy in existing technologies and achieves efficient Chinese word segmentation and part-of-speech tagging, applicable to different fields.

WO2024245081A1PCT designated stage expired Publication Date: 2024-12-05MASHANG CONSUMER FINANCE CO LTD

Patent Information

Application Number
PCT/CN2024/094803
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-29
Filing Date
2024-05-22
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Existing technologies for Chinese word segmentation suffer from low accuracy. In particular, rule-based segmentation methods are highly targeted but labor-intensive, while statistical segmentation methods lack flexibility and accuracy, and machine learning methods have limitations in handling word length.

Method used

By determining the reference labels and inter-point mutual information of the training text, combining BERT or ROBERT models for word segmentation and part-of-speech prediction, using inter-point mutual information to measure the probability of characters forming words, and fusing the reference labels and inter-point mutual information of the training text for model training, the generalization ability of the model is improved.

Benefits of technology

It achieves efficient Chinese word segmentation and part-of-speech tagging, improves the accuracy of new word segmentation, has strong generalization ability, is applicable to different fields, and is not limited by word length.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024094803_05122024_PF_FP_ABST
    Figure CN2024094803_05122024_PF_FP_ABST
Patent Text Reader

Abstract

Disclosed in the present application are a model training method, a text processing method and a related device. The model training method comprises: determining a reference tag corresponding to a training text, the training text comprising a plurality of words, each word consisting of at least one character, and the reference tag being used for marking the position of each character in the training text in the word and the part-of-speech of each word; acquiring pointwise mutual information corresponding to the training text; combining the training text and the pointwise mutual information, and then inputting same into a model for word segmentation and part-of-speech prediction processing, so as to obtain a predicted tag corresponding to the training text, the predicted tag being used for marking the position of each character in the training text when forming a word and the part-of-speech of the word formed on the basis of the character; and, on the basis of the difference between the predicted tag and the reference tag, training the model. Such a way enables models to learn more word forming features during training, so as to improve word segmentation efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Model training methods, text processing methods, and related equipment

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 29, 2023, with application number 202310619279.0 and application name “Training method of text processing model, text processing method and related equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to, but is not limited to, the field of natural language processing technology, and specifically to a model training method, a text processing method, and related equipment. Background Art

[0003] Natural language processing requires computers to accurately extract the meaning of sentences. This process involves understanding syntax, semantic components, semantic results, and context. All of this is based on accurately segmenting a sentence into multiple word combinations. Advanced natural language processing processes, such as personalized recommendations, sentiment analysis, and topic classification, all require highly accurate word segmentation as a prerequisite.

[0004] In Western languages, words are clearly separated by symbols. Natural language processing (NLP) requires only a simple operation to accurately segment an English sentence into several words. Unlike English, Chinese uses words as the smallest unit of meaning, without clear delimiters. Therefore, accurately segmenting text, especially Chinese text, has become a hot research topic in NLP. Technical issues

[0005] To improve word segmentation accuracy, two common approaches are used in related technologies. The first is rule-based word segmentation, where linguists typically construct templates based on word formation principles, combined with semantic and part-of-speech information, and then match them to text. While this method offers high accuracy, it is also highly targeted. Word segmentation rules are generally domain-specific. Applications in different domains require the development of multiple, diverse rules, which is labor-intensive and complex to manually write and maintain. The second is statistical word segmentation, which uses a large corpus of data to calculate statistical features such as word frequency, word probability, left-right adjacency entropy, and adjacency change count to identify words in text. While this method is more flexible and not restricted by domain, it suffers from data sparsity and low accuracy. Given the advantages of machine learning, it can leverage relevant machine learning knowledge to fully leverage the strengths of both statistical and rule-based approaches. However, in practice, this method has been found to have certain limitations on word length and low accuracy. Technical Solutions

[0006] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0007] The present application provides a model training method, a text processing method and related equipment, which can achieve efficient Chinese word segmentation while also achieving part-of-speech tagging, and has a high accuracy rate in segmenting new words.

[0008] In a first aspect, the present application provides a model training method, the method comprising: determining a reference label corresponding to a training text, the training text comprising a plurality of words, each word consisting of at least one character; the reference label being used to mark the position of each character in the training text in the word to which it belongs and the part of speech of each word; obtaining the point-to-point mutual information of two adjacent characters forming a word; concatenating the training text and the point-to-point mutual information and inputting them into the model for word segmentation and part-of-speech prediction processing to obtain a predicted label corresponding to the training text, the predicted label being used to mark the position of each character in the training text when used to form a word and the part of speech of the word formed based on the character; and training the model based on the difference between the predicted label and the reference label.

[0009] In the second aspect, the present application provides a text processing method, which includes: obtaining a text to be processed; performing word segmentation and part-of-speech prediction processing on the text to be processed through a model to obtain a prediction result corresponding to the text to be processed, and the prediction result is used to mark the position of each character in the text to be processed when used to form a word and the part of speech of the word composed of the characters; the model is obtained through the training method of the model as described in the first aspect; performing word segmentation and part-of-speech tagging processing on the text to be processed according to the prediction result to obtain multiple words included in the text to be processed and the part of speech of each word.

[0010] In a third aspect, the present application provides a model training device, which includes: a determination module configured to determine a reference label corresponding to a training text, wherein the training text includes multiple words, and each word consists of at least one character; the reference label is used to mark the position of each character in the training text in the corresponding word and the part of speech of each word; the determination module is also configured to obtain the point-to-point mutual information of two adjacent characters forming a word; the training module is configured to splice the training text and the point-to-point mutual information and input them into the model for word segmentation and part-of-speech prediction processing to obtain a predicted label corresponding to the training text, wherein the predicted label is used to mark the position of each character in the training text when used to form a word and the part of speech of the word composed of the character; the training module is also configured to train the model based on the difference between the predicted label and the reference label.

[0011] In a fourth aspect, the present application provides a text processing device, which includes an acquisition module, a prediction module and a processing module. The acquisition module is configured to acquire a text to be processed. The prediction module is configured to perform word segmentation and part-of-speech prediction processing on the text to be processed through a model to obtain a prediction result corresponding to the text to be processed, and the prediction result is used to mark the position of each character in the text to be processed when it is used to form a word and the part of speech of the word composed of the character; the model is obtained by the training method of the model as described in the first aspect. The processing module is configured to perform word segmentation and part-of-speech tagging processing on the text to be processed according to the prediction result to obtain multiple words included in the text to be processed and the part of speech of each word.

[0012] In a fifth aspect, the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect or the second aspect are implemented.

[0013] In a sixth aspect, the present application provides a computer-readable storage medium storing a program or instruction, which implements the steps of the method described in the first aspect or the second aspect when executed by a processor. Beneficial effects

[0014] In the present application, during the training process of the model, first, a reference label corresponding to a training text is determined, where the training text includes multiple words, and each word consists of at least one character; the reference label is used to mark the position of each character in the training text in the word to which it belongs and the part of speech of each word; then, the point-to-point mutual information of two adjacent characters forming a word is obtained, and the training text and the point-to-point mutual information are concatenated and input into the model for word segmentation and part-of-speech prediction processing to obtain a predicted label corresponding to the training text, and the predicted label is used to mark the position of each character in the training text when used to form a word and the part of speech of the word composed of the characters; finally, the model is trained based on the difference between the predicted label and the reference label. It can be seen that when training the model above, not only the reference labels of the training text are used, but also the point-to-point mutual information of the training text is integrated. The point-to-point mutual information of the training text is used to reflect the probability that two adjacent characters in the training text form a word. The integration of the point-to-point mutual information of the training text and the reference labels to train the model can enable the model to learn the probability between two characters that can form a word, which greatly improves the generalization ability of the model, so that when the model is used to segment any text in the future, the accuracy of new word segmentation can be improved.

[0015] In addition, by using the part of speech of each word in the training text and the position of each character in the training text in the corresponding word as reference labels for the training model, the model can not only learn the segmentation between words and have the ability to segment words, but also learn the part of speech of words in the text. The trained model is used to perform word segmentation and part-of-speech tagging on any text to be processed, thereby achieving efficient Chinese word segmentation and part-of-speech tagging.

[0016] Still other aspects will become apparent upon reading and understanding the accompanying drawings and detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The following is a brief introduction to the drawings required for the description of the embodiments. The drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] FIG1 is a flow chart of a model training method provided in an embodiment of the present application.

[0019] FIG2 is a network structure diagram of a model provided in an embodiment of the present application.

[0020] FIG3 is a flow chart of a text processing method provided in an embodiment of the present application.

[0021] FIG4 is a schematic structural diagram of a model training device provided in an embodiment of the present application.

[0022] FIG5 is a schematic structural diagram of a text processing device provided in an embodiment of the present application.

[0023] FIG6 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. Modes for Carrying Out the Invention

[0024] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. The embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0025] The embodiment of the present application utilizes the reference labels of the training text and the mutual information between points to train the model, so that the model can learn the characteristics that a word should have. In this way, the trained model has a strong generalization ability and can accurately segment the text, especially for new words that are not labeled during the training process. It also has good recognition capabilities.

[0026] Based on this, an embodiment of the present application provides a model training method. Specifically, in the process of training the model, first, a reference label corresponding to the training text is determined, and the training text includes multiple words, and each word consists of at least one character; the reference label is used to mark the position of each character in the training text in the corresponding word and the part of speech of each word; then, the point-to-point mutual information of two adjacent characters forming a word is obtained; the training text and the point-to-point mutual information are spliced ​​and input into the model for word segmentation and part-of-speech prediction processing to obtain the predicted label corresponding to the training text, and the predicted label is used to mark the position of each character in the training text when used to form a word and the part of speech of the word composed of the character; finally, based on the difference between the predicted label and the reference label, the model is trained. It can be seen that when training the model above, not only the reference labels of the training text are used, but also the point-to-point mutual information of the training text is integrated. The point-to-point mutual information of the training text is used to reflect the probability that two adjacent characters in the training text form a word. The integration of the point-to-point mutual information of the training text and the reference labels to train the model can enable the model to learn the probability between two characters that can form a word, which greatly improves the generalization ability of the model, so that when the model is used to segment any text in the future, the accuracy of new word segmentation can be improved.

[0027] In addition, by using the part of speech of each word in the training text and the position of each character in the training text in the corresponding word as reference labels for the training model, the model can not only learn the segmentation between words and have the ability to segment words, but also learn the part of speech of words in the text. The trained model is used to perform word segmentation and part-of-speech tagging on any text to be processed, thereby achieving efficient Chinese word segmentation and part-of-speech tagging.

[0028] FIG1 shows a flow chart of a method for training a model provided in an embodiment of the present application. The method can be executed by an electronic device, such as a terminal device or a server device. In other words, the method can be executed by software or hardware installed on a terminal device or a server device. The server device includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc. The model of the embodiment of the present application can be a text processing model for processing text. As shown in FIG1 , the method may include the following steps.

[0029] S101: Determine a reference label corresponding to a training text.

[0030] Wherein, the training text includes multiple words, and each word consists of at least one character; the reference label is used to mark the position of each character in the training text in the word to which it belongs and the part of speech of each word. It should be noted that the training text mentioned in this step can refer to any one of the sample texts used to train the model. Unless otherwise specified, in the subsequent introduction to the embodiments of this application, taking one training text as an example, the other training texts in the sample text are processed and the model is trained in the same manner as the training text. Wherein, the model can be, but is not limited to, a text prediction model for predicting text, a text recognition model for emotional recognition of text, a text classification model for classifying text, a text generation model for generating text, and other text processing models.

[0031] In one embodiment, the position of a character in the word to which it belongs may include the first position and other positions. In a specific implementation, the position of a character in the word to which it belongs may be represented by a position symbol, for example, B is used to represent the first position or the first character of a word, I represents the second character of a word (two-character word) or any character after the second character (n-character word, n≥3), and O represents a punctuation mark.

[0032] The part of speech of each word can include common nouns, names, adjectives, verbs, etc. In the reference tag, the part of speech of each word can be represented by a part of speech label. Each part of speech corresponds to a label, as shown in Table 1 below.

[0033] In summary, in step S101 , the reference label may include the position symbol of each character in the corresponding word and the label of the part of speech of each word.

[0034] Table 1 Examples of common part-of-speech tags

[0035] Take the training text "The weather is nice today." as an example. Through manual annotation, the text is first segmented to obtain the segmentation result "['today', 'weather', 'nice', '.",]"; then according to the common part-of-speech tags shown in Table 1, the part-of-speech tagging result "['t', 'n', 'a', 'w',]" is obtained. Finally, through data conversion, the reference label corresponding to the training text is obtained, such as: Bt It Bn In Ba Ia Ow, where B represents the first character of a word, I represents the second character of a word (two-word word) or any character after the second character (n-word word, n≥3), and O represents punctuation. The above reference label is only an example and can also be in other forms. This application does not make specific restrictions.

[0036] S102: Obtain the pointwise mutual information of a word formed by two adjacent characters.

[0037] Pointwise Mutual Information (PMI) is used to measure the correlation between two events. In the embodiments of this application, pointwise mutual information is used to measure the probability that two adjacent characters among multiple characters of the training text form a word. Its calculation formula is:

[0038] Where x and y respectively represent two adjacent characters, P(x, y) represents the probability that adjacent characters x and y form a word, and P(x) and P(y) respectively represent the probabilities of the occurrence of characters x and y. Taking the two characters "zhi" and "hu" as an example, assuming that in a 500-word sample, "zhi" appears 20 times, "hu" appears 10 times, and "zhihu" appears 5 times, then the probability of "zhi" appearing is 0.04, the probability of "hu" appearing is 0.02, and the probability of "zhi" and "hu" appearing together is 0.01. Then the pointwise mutual information of "zhi" and "hu" is 1.09.

[0039] The greater the pointwise mutual information of two adjacent characters, the greater the probability that they form a word. For example, in the aforementioned training text "Today's weather is nice.", the pointwise mutual information values between each pair of adjacent characters are 0.23, 0.14, 0.12, 0.039, 0.29, and 0.095 respectively.

[0040] S103: Concatenate the training text and the pointwise mutual information and input them into the model for word segmentation and词性 prediction processing to obtain the prediction labels corresponding to the training text. The prediction labels are used to mark the positions of each character in the training text when forming words and the词性 of the words formed based on the characters.

[0041] Specifically, for example, concatenate the aforementioned training text "Today's weather is nice." with the vector "0.23, 0.14, 0.12, 0.039, 0.29, 0.095" composed of its corresponding pointwise mutual information as the input of the model to obtain the prediction labels corresponding to the training text, such as "B-t I-t B-n I-n B-a I-a O-w", where the meanings of each symbol are the same as those in the aforementioned reference labels.

[0042] In one embodiment, concatenating the training text and the inter-point mutual information and inputting them into the model may include: adding a start marker before the training text, connecting the training text and the inter-point mutual information via a sentence separator to obtain concatenated information; and inputting the concatenated information into the model. The start marker is represented by [CLS], and the sentence separator can be represented by [SEP]. For example, if the training text is "The weather is good today" and the inter-point mutual information is represented by "0.23, 0.14, 0.12, 0.039, 0.29, 0.095", the concatenated information obtained by concatenating the training text and the inter-point mutual information can be represented as "[CLS]The weather is good today [SEP] 0.23, 0.14, 0.12, 0.039, 0.29, 0.095".

[0043] In one implementation, the model includes: a trainer and a classifier. In the embodiment of the present application, in step S103, the training text and the mutual information between points can be spliced ​​and then input into the trainer for encoding processing to obtain a first encoding vector corresponding to the training text and a second encoding vector corresponding to the mutual information between points respectively; then, the first encoding vector and the second encoding vector are superimposed and input into the classifier for mapping processing to obtain a predicted label corresponding to the training text, thereby realizing word segmentation and part of speech prediction of the spliced ​​information.

[0044] Figure 2 shows a network structure diagram of a model provided by an embodiment of the present application. The input of the model can be a text sequence in a predetermined format: [CLS] training text [SEP] point-to-point mutual information, where [CLS] is the start symbol and [SEP] is the sentence separator. Taking the training text "The weather is good today." as an example, the training text can be expressed as "The weather is good today.", and the point-to-point mutual information can be expressed as "0.23, 0.14, 0.12, 0.039, 0.29, 0.095". Each value in "0.23, 0.14, 0.12, 0.039, 0.29, 0.095" corresponds to the probability that two adjacent characters in the training text form a word.

[0045] Optionally, the trainer can include a BERT model or a ROBERT model. The BERT model or ROBERT model is a pre-trained language model that uses a bidirectional encoder representation of a transformer. The pre-trained BERT representation can be fine-tuned through an additional output layer and is suitable for building models for a wide range of tasks.

[0046] After the training text and the point-to-point mutual information are concatenated, they are passed through a trainer such as BERT or ROBERT to obtain a first encoding vector vec1 corresponding to the training text and a second encoding vector vec2 corresponding to the point-to-point mutual information. Vec1 and vec2 are then concatenated or added together to obtain a vector vec, which is input into a classifier to obtain a predicted label for each character in the training text. This embodiment of the present application utilizes point-to-point mutual information to measure the probability of two adjacent characters forming a word. By superimposing point-to-point mutual information values, the word segmentation performance of the model is improved.

[0047] In an embodiment of the present application, a classifier is used to perform word segmentation and part-of-speech prediction on the training text encoded by the trainer to obtain a predicted label. Optionally, the classifier can be a decision tree classifier, a Bayesian classifier, a nearest neighbor classifier, a support vector machine classifier, a logistic regression classifier, etc., and this application does not make specific restrictions. Among them, softmax can be selected as the activation function of the classifier, and the normalized value of each type of data is calculated to obtain the predicted label.

[0048] The training method of the model provided in the embodiment of the present application is not restricted by the field, is easy to expand and has good portability. At the same time, due to the generalization ability of the model, it can have good recognition of new words (words that have not appeared in the labeled data).

[0049] S104: Train a model based on the difference between the predicted label and the reference label.

[0050] The difference between the predicted label and the reference label can reflect which words or parts of speech the model mispredicted during the word segmentation and part-of-speech prediction process, and the model parameters can then be adjusted to avoid the next error. Specifically, in step S104, the model can determine a loss function based on the difference between the predicted label and the reference label; then, the model parameters of the model can be adjusted in a direction that reduces the value of the loss function to achieve training of the model. For example, the model parameters of the model are adjusted until the loss function value is less than a preset threshold, or the number of training times reaches a preset value, resulting in a trained model.

[0051] In the embodiment of the present application, because the predicted label marks the position of each character in the training text when it is used to form a word and the part of speech of the word composed of the characters, the trained model can not only learn the segmentation between words and have the ability to segment words, but also learn the part of speech of each character in the word composed of the characters. In the model recognition stage, it can be used to perform word segmentation and part-of-speech tagging on any text to be processed, and the entire process does not involve the calculation of probability, entropy, etc. in related statistical methods, and is highly efficient. In addition, since there is no limit on the length of the words in the training text, the obtained model can be used to determine words of any length, and the words are only related to the part of speech. As long as the training text contains words of various parts of speech, then for any new word, its part of speech must belong to the part-of-speech set contained in the training text, and it can be identified with a high accuracy rate.

[0052] In an embodiment of the present application, during the training process of the model, first, a reference label corresponding to a training text is determined, where the training text includes multiple words, and each word consists of at least one character; the reference label is used to mark the position of each character in the training text in the word to which it belongs and the part of speech of each word; then, the point-to-point mutual information of a word composed of two adjacent characters is obtained, and the point-to-point mutual information is used to represent the probability that two adjacent characters in multiple characters of the training text form a word; the training text and the point-to-point mutual information are concatenated and input into the model for word segmentation and part-of-speech prediction processing to obtain a predicted label corresponding to the training text, and the predicted label is used to mark the position of each character in the training text when used to form a word and the part of speech of the word composed of the character; finally, based on the difference between the predicted label and the reference label, the model is trained. It can be seen that when training the model above, not only the reference labels of the training text are used, but also the point-to-point mutual information of the training text is integrated. The point-to-point mutual information of the training text is used to reflect the probability that two adjacent characters in the training text form a word. The integration of the point-to-point mutual information of the training text and the reference labels to train the model can enable the model to learn the probability between two characters that can form a word, which greatly improves the generalization ability of the model, so that when the model is used to segment any text in the future, the accuracy of new word segmentation can be improved.

[0053] In addition, by using the part of speech of each word in the training text and the position of each character in the training text in the corresponding word as reference labels for the training model, the model can not only learn the segmentation between words and have the ability to segment words, but also learn the part of speech of words in the text. The trained model is used to perform word segmentation and part-of-speech tagging on any text to be processed, thereby achieving efficient Chinese word segmentation and part-of-speech tagging.

[0054] FIG3 shows a flow chart of a text processing method provided by an embodiment of the present application. The method can be executed by an electronic device, such as a terminal device or a server device. In other words, the method can be executed by software or hardware installed on the terminal device or the server device. The server device includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. As shown in FIG3 , the method may include the following steps.

[0055] S301: Obtain the text to be processed.

[0056] Among them, the text to be processed is the text that needs to be segmented. The text to be processed can be text generated in any application scenario that requires segmentation. For example, the text to be processed can be text in a personalized recommendation scenario. In the personalized recommendation scenario, the text to be processed can be a user's historical browsing record text collected from an application or website that requires personalized recommendation; after the historical browsing record text is segmented, the keywords that the user is interested in can be extracted, and then news or products can be pushed to the user based on the extracted keywords. For another example, the text to be processed can also be text in a sentiment analysis scenario. In the sentiment analysis scenario, the text to be processed refers to the interactive text communicated between the customer and the customer service. After the interactive text is segmented, the customer's emotional tendency is analyzed based on the extracted segmented words. For customers with a tendency to complain, manual intervention is performed in a timely manner to reduce the complaint rate. For another example, the text to be processed can also be text in other scenarios that require segmentation, such as topic classification and public opinion analysis. The embodiments of this application will not be expanded one by one.

[0057] S302: Perform word segmentation and part-of-speech prediction on the text to be processed through the model to obtain a prediction result corresponding to the text to be processed.

[0058] The prediction results are used to mark the position of each character in the text to be processed when it is used to form a word and the part of speech of the word composed of the characters; the model is obtained through the training method embodiment of the model shown in Figure 2. For details, please refer to the introduction of the aforementioned embodiment, which will not be repeated here.

[0059] Specifically, for example, the text to be processed is "I will play basketball tomorrow." The model is used to predict the label of each character in the text to be processed. Correspondingly, it is assumed that the prediction result can be expressed as: "Br, Bt, It, Bv, Bn, In, Ow".

[0060] S303: Perform word segmentation and part-of-speech tagging on the text to be processed according to the prediction result to obtain multiple words included in the text to be processed and the part of speech of each word.

[0061] Based on the prediction results from the previous steps, this step yields the following: 1. The word segmentation result for the text to be processed is "I / tomorrow / play / basketball / ." 2. The corresponding parts of speech for each word are "r / t / v / n / w" respectively. At this point, the model has completed word segmentation and part-of-speech tagging for the text to be processed.

[0062] In one implementation, the text to be processed can be obtained in response to a key information extraction event. After obtaining the multiple words included in the text to be processed and the part of speech of each word, a key information extraction event can be executed based on the multiple words included in the text to be processed to output the key information corresponding to the text to be processed.

[0063] Faced with a vast amount of information, it's often difficult to fully absorb it. Therefore, it's necessary to filter out interesting or representative information. For example, an online store selling a new air conditioner might segment hundreds of thousands of reviews and extract key information, such as "discount," "electrical leakage," "bad attitude," "slow delivery," or "water leakage," to adjust sales strategies. Another example is document topic extraction, where keywords can be extracted to summarize the main theme of a long article.

[0064] In one implementation, based on multiple words included in the text to be processed, a key information extraction event is executed to output key information corresponding to the text to be processed, including: calculating the word frequency (TF value) and inverse text frequency index (IDF value) of each word in the text to be processed respectively; calculating the importance (TF-IDF) value of each word based on the word frequency and inverse text frequency index of each word and the importance calculation formula; selecting words whose importance values ​​meet the threshold to generate the key information corresponding to the text to be processed.

[0065] The Term Frequency-Inverse Document Frequency (TF-IDF) algorithm is a statistically based calculation method used to assess the importance of a term in a document. The importance of a term increases proportionally with its occurrence in a document, but decreases inversely with its frequency in the corpus. Term Frequency (TF) refers to the frequency with which a given term appears in a document, specifically the ratio of the number of times term w appears in document d (count(w, d)) to the total number of terms in document d (size(d)). Inverse Document Frequency (IDF) is a measure of the general importance of a term. The IDF of a particular term can be calculated by dividing the total number of documents by the number of documents containing the term and taking the logarithm of the quotient. This is the logarithm of the ratio of the total number of documents n to the number of documents in which term w appears (docs(w, D)). The TF-IDF value for each term is then calculated by multiplying the TF and IDF. A high term frequency within a specific document combined with a low term frequency across the entire document collection results in a highly weighted TF-IDF. Therefore, TF-IDF tends to filter out common words and retain important words. Finally, key information corresponding to the text to be processed is generated by selecting words whose TF-IDF values ​​meet the threshold.

[0066] In the embodiment of the present application, when training the model, not only the reference labels of the training text are used, but also the point-to-point mutual information of the training text is integrated. The point-to-point mutual information of the training text is used to reflect the probability that two adjacent characters in the training text form a word. The integration of the point-to-point mutual information of the training text and the reference labels to train the model can enable the model to learn what the probability is between two characters that can form a word, greatly improving the generalization ability of the model, so that when the model is subsequently used to segment any text, the accuracy of the segmentation of new words can be improved. In addition, by using the part of speech of each word in the training text and the position of each character in the training text in the word to which it belongs as the reference label of the training model, the model can learn the segmentation between words and have the ability to segment words, and can also learn the part of speech of words in the text. The model after training is used to perform word segmentation and part-of-speech tagging on any text to be processed, thereby achieving efficient Chinese word segmentation while achieving part-of-speech tagging. The model obtained by the above can improve the accuracy of word segmentation of the text to be processed.

[0067] FIG4 shows a schematic structural diagram of a model training device provided in an embodiment of the present application. As shown in FIG4 , the device 400 includes: a determination module 410 and a training module 420 .

[0068] The determination module 410 may be configured to determine a reference label corresponding to a training text, wherein the training text includes a plurality of words, each word consisting of at least one character; the reference label is used to mark the position of each character in the training text within the corresponding word and the part of speech of each word; the determination module 410 may also be configured to obtain point-to-point mutual information of a word consisting of two adjacent characters, wherein the point-to-point mutual information is used to represent the probability that two adjacent characters in the plurality of characters in the training text form a word;

[0069] The training module 420 can be configured to concatenate the training text and the mutual information between the points and input them into the model for word segmentation and part-of-speech prediction processing to obtain a predicted label corresponding to the training text, wherein the predicted label is used to mark the position of each character in the training text when used to form a word and the part of speech of the word composed of the character; the training module 420 can also be configured to train the model based on the difference between the predicted label and the reference label; wherein the model after training is used to perform word segmentation and part-of-speech tagging processing on any text to be processed.

[0070] In one implementation, the model includes a trainer and a classifier. When the training module 420 splices the training text and the point-to-point mutual information and inputs them into the model for word segmentation and part-of-speech prediction processing to obtain the predicted label corresponding to the training text, the following steps are performed: the training text and the point-to-point mutual information are spliced ​​and input into the trainer for encoding processing to obtain a first encoding vector corresponding to the training text and a second encoding vector corresponding to the point-to-point mutual information; the first encoding vector and the second encoding vector are superimposed and input into the classifier for mapping processing to obtain the predicted label corresponding to the training text.

[0071] In one implementation, when training the model based on the difference between the predicted label and the reference label, the training module 420 performs the following steps: determining a loss function based on the difference between the predicted label and the reference label; and adjusting the model parameters of the model in a direction to reduce the value of the loss function.

[0072] In one implementation, when the training module 420 splices the training text and the mutual information between points and inputs them into the model, it performs the following steps: adding a start marker before the training text, and connecting the training text and the mutual information between points through a sentence separator to obtain the spliced ​​information; and inputting the spliced ​​information into the model.

[0073] In an embodiment of the present application, during the training process of the model, first, a reference label corresponding to a training text is determined, where the training text includes multiple words, and each word consists of at least one character; the reference label is used to mark the position of each character in the training text in the word to which it belongs and the part of speech of each word; then, the point-to-point mutual information of two adjacent characters forming a word is obtained, and the point-to-point mutual information is used to represent the probability that two adjacent characters in multiple characters of the training text form a word; the training text and the point-to-point mutual information are concatenated and input into the model for word segmentation and part-of-speech prediction processing to obtain a predicted label corresponding to the training text, and the predicted label is used to mark the position of each character in the training text when used to form a word and the part of speech of the word composed of the character; finally, the model is trained based on the difference between the predicted label and the reference label. It can be seen that when training the model above, not only the reference labels of the training text are used, but also the point-to-point mutual information of the training text is integrated. The point-to-point mutual information of the training text is used to reflect the probability that two adjacent characters in the training text form a word. The integration of the point-to-point mutual information of the training text and the reference labels to train the model can enable the model to learn the probability between two characters that can form a word, which greatly improves the generalization ability of the model, so that when the model is used to segment any text in the future, the accuracy of new word segmentation can be improved.

[0074] In addition, by using the part of speech of each word in the training text and the position of each character in the training text in the corresponding word as reference labels for the training model, the model can not only learn the segmentation between words and have the ability to segment words, but also learn the part of speech of words in the text. The trained model is used to perform word segmentation and part-of-speech tagging on any text to be processed, thereby achieving efficient Chinese word segmentation and part-of-speech tagging.

[0075] FIG5 shows a schematic structural diagram of a text processing device provided in an embodiment of the present application. As shown in FIG5 , the device 500 includes: an acquisition module 510 , a prediction model 520 and a processing module 530 .

[0076] The acquisition module 510 may be configured to acquire the text to be processed;

[0077] The prediction module 520 may be configured to perform word segmentation and part-of-speech prediction processing on the text to be processed using a model to obtain a prediction result corresponding to the text to be processed, wherein the prediction result is used to mark the position of each character in the text to be processed when used to form a word and the part of speech of the word formed based on the character; the model is obtained by executing any of the aforementioned model training methods;

[0078] The processing module 530 may be configured to perform word segmentation and part-of-speech tagging on the text to be processed according to the prediction result, so as to obtain a plurality of words included in the text to be processed and the part of speech of each word.

[0079] In one implementation, the text to be processed is obtained in response to a key information extraction event, and the processing module 530 can also be configured to execute the key information extraction event based on multiple words included in the text to be processed and output the key information corresponding to the text to be processed.

[0080] In one implementation, the processing module 530 can be configured to respectively calculate the word frequency and inverse text frequency index of each word in the text to be processed; calculate the importance value of each word based on the word frequency and inverse text frequency index of each word and the importance calculation formula; and select words whose importance values ​​meet the threshold to generate key information corresponding to the text to be processed.

[0081] In the embodiment of the present application, when training the model, not only the reference labels of the training text are used, but also the point-to-point mutual information of the training text is integrated. The point-to-point mutual information of the training text is used to reflect the probability that two adjacent characters in the training text form a word. The integration of the point-to-point mutual information of the training text and the reference labels to train the model can enable the model to learn what the probability is between two characters that can form a word, greatly improving the generalization ability of the model, so that when the model is subsequently used to segment any text, the accuracy of the segmentation of new words can be improved. In addition, by using the part of speech of each word in the training text and the position of each character in the training text in the word to which it belongs as the reference label of the training model, the model can learn the segmentation between words and have the ability to segment words, and can also learn the part of speech of words in the text. The model after training is used to perform word segmentation and part-of-speech tagging on any text to be processed, thereby achieving efficient Chinese word segmentation while achieving part-of-speech tagging. The model obtained by the above can improve the accuracy of word segmentation of the text to be processed.

[0082] FIG6 shows a schematic diagram of the hardware structure of an electronic device that implements an embodiment of the present application. Referring to the figure, at the hardware level, the electronic device includes a processor. Optionally, the electronic device may also include an internal bus, a network interface, and a memory. Among them, the memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0083] The processor, network interface, and memory can be interconnected via an internal bus, such as an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. These buses can be classified as address buses, data buses, and control buses. For ease of illustration, the figure uses only one bidirectional arrow, but this does not imply that there is only one bus or only one type of bus.

[0084] The memory is used to store programs. Specifically, the program may include program code, which includes computer operating instructions. The memory may include internal memory and non-volatile memory, and provides instructions and data to the processor.

[0085] The processor reads the corresponding computer program from the non-volatile memory into the internal memory and then runs it, logically forming a device for locating the target user. The processor executes the program stored in the memory and is specifically configured to perform the steps of the model training method described in at least one of the above method embodiments or the steps of the text processing method described in at least one of the above method embodiments.

[0086] The methods disclosed in the embodiments shown in the flowcharts of the present application can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, the steps of the above methods can be completed by hardware integrated logic circuits in the processor or by software instructions. The above processors can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present application can be directly implemented as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0087] The electronic device can also execute the model training method described in at least one embodiment of the above method embodiments or the various steps of the text processing method described in at least one embodiment of the above method embodiments, and can achieve the same technical effect as the above method embodiments, which will not be repeated here.

[0088] Of course, in addition to software implementation, the electronic device of this application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0089] An embodiment of the present application also proposes a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple application programs, the electronic device executes the various steps of the model training method described in at least one embodiment of the above-mentioned method embodiments or the various steps of the text processing method described in at least one embodiment of the above-mentioned method embodiments.

[0090] The computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0091] Optionally, an embodiment of the present application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the steps of at least one embodiment of the above-mentioned method embodiments are implemented.

[0092] In short, the above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

[0093] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0094] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0095] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0096] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

Claims

1. A model training method, the method comprising: Determine a reference label corresponding to a training text, wherein the training text includes a plurality of words, each word consists of at least one character; the reference label is used to mark the position of each character in the training text in the corresponding word and the part of speech of each word; Obtaining the mutual information between points where two adjacent characters form a word; The training text and the point-to-point mutual information are spliced ​​and input into the model for word segmentation and part-of-speech prediction processing to obtain a prediction label corresponding to the training text, wherein the prediction label is used to mark the position of each character in the training text when it is used to form a word and the part-of-speech of the word formed based on the character; The model is trained based on the difference between the predicted labels and the reference labels.

2. According to the method of claim 1, the model includes a trainer and a classifier, and the concatenation of the training text and the point-to-point mutual information is input into the model for word segmentation and part-of-speech prediction to obtain a predicted label corresponding to the training text, including: The training text and the point-to-point mutual information are concatenated and input into the trainer for encoding, so as to obtain a first encoding vector corresponding to the training text and a second encoding vector corresponding to the point-to-point mutual information respectively; The first encoding vector and the second encoding vector are superimposed and input into the classifier for mapping processing to obtain a predicted label corresponding to the training text.

3. The method according to claim 2, wherein training the model based on the difference between the predicted label and the reference label comprises: determining a loss function based on a difference between the predicted label and the reference label; The model parameters of the model are adjusted in a direction that reduces the value of the loss function.

4. The method according to claim 1, wherein the step of concatenating the training text and the point-to-point mutual information and inputting the concatenated information into the model comprises: Adding a start marker before the training text, and connecting the training text and the point-to-point mutual information through a sentence separator to obtain concatenated information; Input the stitched information into the model.

5. A text processing method, comprising: Get the text to be processed; The model is used to perform word segmentation and part-of-speech prediction processing on the text to be processed, and obtains a prediction result corresponding to the text to be processed, wherein the prediction result is used to mark the position of each character in the text to be processed when it is used to form a word and the part-of-speech of the word composed of the character; the model is obtained by the training method of the model according to any one of claims 1 to 4; The text to be processed is segmented and marked with parts of speech according to the prediction result to obtain a plurality of words included in the text to be processed and the part of speech of each word.

6. The method according to claim 5, wherein the text to be processed comprises: Historical browsing record text for personalized recommendations, communication interaction text for sentiment analysis, text for topic classification, and text for public opinion analysis.

7. According to the method of claim 5, the text to be processed is obtained in response to a key information extraction event, and after obtaining a plurality of words included in the text to be processed and the part of speech of each word, the method further comprises: Based on the multiple words included in the text to be processed, the key information extraction event is executed to output the key information corresponding to the text to be processed.

8. The method according to claim 7, wherein the step of performing the key information extraction event based on the plurality of words included in the text to be processed and outputting the key information corresponding to the text to be processed comprises: Calculating the word frequency and inverse text frequency index of each word in the text to be processed respectively; Calculate the importance value of each word based on the word frequency and inverse text frequency index of each word and the importance calculation formula; Select words whose importance values ​​meet a threshold value to generate key information corresponding to the text to be processed.

9. A model training device, comprising: A determination module, configured to determine a reference label corresponding to a training text, wherein the training text includes a plurality of words, each word consists of at least one character; the reference label is used to mark the position of each character in the training text in the corresponding word and the part of speech of each word; The determination module is further configured to obtain the point-to-point mutual information of two adjacent characters forming a word; A training module is configured to splice the training text and the point-to-point mutual information and input them into a model for word segmentation and part-of-speech prediction processing to obtain a prediction label corresponding to the training text, wherein the prediction label is used to mark the position of each character in the training text when it is used to form a word and the part of speech of the word formed based on the character; The training module is further configured to train the model based on the difference between the predicted label and the reference label.

10. The device according to claim 9, wherein the model comprises a trainer and a classifier, and the training module performs the following steps when splicing the training text and the point-to-point mutual information and inputting them into the model for word segmentation and part-of-speech prediction processing to obtain the predicted label corresponding to the training text: The training text and the point-to-point mutual information are concatenated and input into the trainer for encoding processing to obtain a first encoding vector corresponding to the training text and a second encoding vector corresponding to the point-to-point mutual information, respectively; the first encoding vector and the second encoding vector are superimposed and input into the classifier for mapping processing to obtain a predicted label corresponding to the training text.

11. The apparatus according to claim 10, wherein the training module performs the following steps when training the model based on the difference between the predicted label and the reference label: A loss function is determined based on the difference between the predicted label and the reference label; and a model parameter of the model is adjusted in a direction to reduce the value of the loss function.

12. The device according to claim 9, wherein when the training text and the point-to-point mutual information are concatenated and input into the model, the model performs the following steps: A start marker is added before the training text, and the training text and the point-to-point mutual information are connected via a sentence separator to obtain concatenated information; and the concatenated information is input into the model.

13. A text processing device, comprising: The acquisition module is set to acquire the text to be processed; A prediction module, configured to perform word segmentation and part-of-speech prediction processing on the text to be processed through a model to obtain a prediction result corresponding to the text to be processed, wherein the prediction result is used to mark the position of each character in the text to be processed when used to form a word and the part-of-speech of the word composed of the character; the model is obtained by the training method of the model according to any one of claims 1 to 4; The processing module is configured to perform word segmentation and part-of-speech tagging processing on the text to be processed according to the prediction result, so as to obtain multiple words included in the text to be processed and the part of speech of each word.

14. The device according to claim 13, wherein the text to be processed is obtained in response to a key information extraction event, and the processing module is further configured to: Based on the multiple words included in the text to be processed, the key information extraction event is executed to output the key information corresponding to the text to be processed.

15. The device according to claim 14, wherein the processing module is further configured to: Calculating the word frequency and inverse text frequency index of each word in the text to be processed respectively; Calculate the importance value of each word based on the word frequency and inverse text frequency index of each word and the importance calculation formula; Select words whose importance values ​​meet a threshold value to generate key information corresponding to the text to be processed.

16. An electronic device comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method according to any one of claims 1-4 or 5-8.

Citation Information

Patent Citations

  • Model training method and device, and method and device for realizing text processing

    CN111160026A

  • Keyword extraction method and device and electronic equipment

    CN114692626A

  • New word discovery method, device and equipment and computer readable storage medium

    CN115879452A

  • Annotated text data expanding method, annotated text data expanding computer-readable storage medium, annotated text data expanding device, and text classification model training method

    US20190317986A1

Cited By

  • Sensitive information processing method and related device

    CN119808162A

  • Data cleaning method

    CN119884611A

  • Chinese smoothness assessment method and apparatus, electronic device, and readable medium

    WO2026123854A1