Pre-training language model training method, text sentiment classification method and device
By combining word segmentation with occlusion and sentence relationship tasks to train a pre-trained language model, the problem of inaccurate semantic representation between sentences was solved, and accurate classification of text sentiment and identification of user complaint tendencies were achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- MASHANG CONSUMER FINANCE CO LTD
- Filing Date
- 2022-11-16
- Publication Date
- 2026-06-02
AI Technical Summary
Existing pre-trained language models cannot accurately represent the semantics between sentences, resulting in low accuracy in text sentiment classification and an inability to effectively predict users' complaint tendencies.
By acquiring the sequence text information of the training samples, word segmentation is performed to obtain the basic feature vector and semantic feature vector of each word. Combined with the occlusion task and sentence relationship task in the pre-trained language model, the input text vector is generated, and the pre-trained language model is trained based on the loss function.
It improves the semantic representation ability of pre-trained language models, enabling them to accurately predict text sentiment classification and thus identify user complaint tendencies.
Smart Images
Figure CN116127062B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of natural language processing technology, and in particular to a training method for a pre-trained language model, a text sentiment classification method, and an apparatus. Background Technology
[0002] In recent years, user reviews have become increasingly important for product marketing. Predicting user complaint trends can help avoid negative public perception of the product. Currently, user review analysis often utilizes pre-trained language models. The development of pre-trained language models has elevated research in the field of Natural Language Processing (NLP) to a new level, enabling the learning of general language representations from massive amounts of corpora and significantly improving downstream task capabilities without requiring the model to be learned from scratch.
[0003] However, existing pre-trained language models cannot accurately represent the semantics between sentences, resulting in low accuracy in sentiment classification of text data and consequently failing to predict user complaint tendencies. Therefore, a training method for pre-trained language models is needed to improve the semantic representation between sentences, thereby accurately predicting text sentiment classification and enabling the identification of user complaint tendencies based on text sentiment. Summary of the Invention
[0004] This application provides a training method and apparatus for a pre-trained language model, which can improve the semantic representation between sentences, thereby accurately predicting the classification of text sentiment, and thus enabling the identification of user complaint tendencies based on text sentiment.
[0005] Firstly, this application provides a method for training a pre-trained language model, comprising:
[0006] Obtain training samples and the corresponding sequence text information of the training samples;
[0007] The sequence text information is segmented to obtain multiple segments included in the training sample;
[0008] Obtain the basic feature vector of each word in the multiple word segments, and obtain the semantic feature vector of each word segment. Then, concatenate the basic feature vector of each word segment and the semantic feature vector of each word segment to obtain the input text vector.
[0009] Based on the occlusion task and sentence relationship task in the pre-trained language model, the input text vector is processed to obtain the loss function of the pre-trained language model, and the pre-trained language model is trained based on the loss function.
[0010] Secondly, this application provides a text sentiment classification method, including:
[0011] Obtain the comment text to be classified into sentiment and the corresponding sequence text information of the comment text;
[0012] The sequence text information is segmented to obtain multiple words included in the comment text;
[0013] Obtain the basic feature vector of each word in the multiple word segments, and obtain the semantic feature vector of each word segment. Then, concatenate the basic feature vector of each word segment and the semantic feature vector of each word segment to obtain the input text vector.
[0014] The input text vector is input into a sentiment recognition model for sentiment recognition to obtain the sentiment category reflected by the comment text; wherein, the sentiment recognition model includes: a pre-trained language model and an output layer; the pre-trained language model is used to determine the sentiment category of the comment text based on the occlusion task and sentence relationship task in the sentiment recognition model, and the output layer is used to output the sentiment category; the pre-trained language model is trained by the method described in the first aspect.
[0015] Thirdly, this application provides a training apparatus for a pre-trained language model, the apparatus comprising:
[0016] The acquisition unit is used to acquire training samples and the corresponding sequence text information of the training samples;
[0017] The processing unit is used to perform word segmentation on the sequence text information to obtain multiple words included in the training sample;
[0018] The acquisition unit is also used to acquire the basic feature vector of each word in the plurality of word segments, and to acquire the semantic feature vector of each word segment;
[0019] The concatenation unit is used to concatenate the basic feature vector of each word and the semantic feature vector of each word to obtain the input text vector;
[0020] The processing unit is further configured to process the input text vector based on the occlusion task and the sentence relationship task in the pre-trained language model, respectively, to obtain the loss function of the pre-trained language model, and to train the pre-trained language model based on the loss function.
[0021] Fourthly, this application provides a text sentiment classification device, the device comprising:
[0022] The acquisition unit is used to acquire the comment text to be classified into sentiment and the corresponding sequence text information of the comment text;
[0023] The processing unit is used to perform word segmentation on the sequence text information to obtain multiple words included in the comment text;
[0024] The concatenation unit is used to obtain the basic feature vector of each word in the multiple word segments, and to obtain the semantic feature vector of each word segment. The basic feature vector and the semantic feature vector of each word segment are concatenated to obtain the input text vector.
[0025] A determining unit is used to input the input text vector into a sentiment recognition model for sentiment recognition to obtain the sentiment category reflected by the comment; wherein, the sentiment recognition model includes: a pre-trained language model and an output layer; the pre-trained language model is used to determine the sentiment category of the comment text based on the occlusion task and sentence relationship task in the sentiment recognition model, and the output layer is used to output the sentiment category; the pre-trained language model is trained by the method described in the first aspect.
[0026] Fifthly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;
[0027] The memory stores computer-executed instructions;
[0028] The processor executes computer execution instructions stored in the memory to implement the method as described in the first or second aspect.
[0029] In a sixth aspect, this application provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement the method described in the first or second aspect.
[0030] In a seventh aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first or second aspect.
[0031] This application provides a training method for a pre-trained language model, a text sentiment classification method, and an apparatus. When training the pre-trained language model, the method first acquires training samples and the corresponding sequence text information. The sequence text information is then segmented to obtain multiple segments included in the training samples. Next, the basic feature vector of each segment and the semantic feature vector of each segment are acquired, and these are concatenated to obtain an input text vector. Further, based on the occlusion task and sentence relationship task in the pre-trained language model, the input text vector is processed to obtain the loss function of the pre-trained language model, and the pre-trained language model is trained based on the loss function. This technical solution improves the semantic representation of each segmented word by acquiring its semantic feature vector. Furthermore, after generating input text vectors from the segmented words, it also enhances the semantic representation of the input text vectors. By using the input text vectors to train a pre-trained language model, the semantic representation capability of the pre-trained language model is improved, enabling the pre-trained language model to accurately predict the classification of text sentiment. This allows for accurate identification of user complaint tendencies based on text sentiment. Attached Figure Description
[0032] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0033] Figure 1 This is a flowchart illustrating a training method for a pre-trained language model according to Embodiment 1 of this application;
[0034] Figure 2a This is a schematic diagram of the architecture of a pre-trained language model provided according to Embodiment 1 of this application;
[0035] Figure 2b This is a flowchart illustrating a training method for a pre-trained language model according to Embodiment 2 of this application;
[0036] Figure 3 This is a flowchart illustrating a text sentiment classification method according to Embodiment 3 of this application;
[0037] Figure 4 This is a schematic diagram of a training device for a pre-trained language model according to Embodiment 4 of this application;
[0038] Figure 5 This is a schematic diagram of a text sentiment classification device according to Embodiment Six of this application;
[0039] Figure 6 This is a block diagram illustrating an electronic device according to an exemplary embodiment.
[0040] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0041] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0042] The training method for the pre-trained language model provided in this application aims to solve the above-mentioned technical problems in the prior art.
[0043] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.
[0044] Figure 1 This is a flowchart illustrating a training method for a pre-trained language model according to Embodiment 1 of this application. Figure 1 The training method for the pre-trained language model shown can be executed by an electronic device, which may include a terminal or a server. The terminal may include a mobile phone, laptop, smart interactive device, or in-vehicle terminal, etc., and the server may include a standalone physical server, a server cluster consisting of multiple servers, or a cloud server capable of cloud computing. Example 1 includes the following steps:
[0045] S101. Obtain the training samples and the corresponding sequence text information.
[0046] In this embodiment, the training samples include a large number of users' historical comment data samples, which are selected from the original training sample library. Sequence text information is text information generated by linearly arranging the training samples. Sequence text information can be read by a computer, and each training sample will generate a sequence text information.
[0047] S102. Perform word segmentation on the sequence text information to obtain multiple words included in the training samples.
[0048] In this embodiment, the technology adopted for word segmentation processing can be a word segmentation method based on a dictionary; it can also be a word segmentation method based on statistics; or a word segmentation method based on rules can also be adopted.
[0049] Among them, the algorithms specifically implemented for the word segmentation method based on a dictionary can be the maximum matching method and the minimum segmentation method. This method is relatively simple and has a relatively high word segmentation efficiency. However, for a relatively large number of sequence text information, it is not very easy to implement.
[0050] The word segmentation method based on statistics is based on the statistical information of words and characters, such as applying the information between adjacent characters, word frequency, and corresponding co-occurrence information, etc. to word segmentation. The word segmentation method based on rules is to perform word segmentation according to preset rules.
[0051] In this embodiment, for example, if the training sample is "Spring river water warms, ducks know first", then after word segmentation processing, the multiple segmented words obtained are "Spring river", "Water warms", "Ducks", "Know first".
[0052] S103. Obtain the basic feature vector of each segmented word among the multiple segmented words, and obtain the sememe feature vector of each segmented word, and splice the basic feature vector of each segmented word and the sememe feature vector of each segmented word to obtain the input text vector.
[0053] In this embodiment, perform dimensional normalization processing on the basic feature vector and sememe feature vector of each segmented word, and splice the normalized basic feature vector and the normalized sememe feature vector to further determine the input text vector.
[0054] In one embodiment, the basic feature vector of each segmented word includes a position feature vector, a chunk feature vector, and a word feature vector; assuming that the multiple segmented words include a target segmented word, taking the target segmented word as an example, obtaining the basic feature vector of each segmented word can include: looking up the word feature vector corresponding to the target segmented word in the preset word vector table according to the mapping relationship between the words and the word feature vectors in the preset word vector table; determining the position feature vector of the target segmented word and the chunk feature vector of the target segmented word based on the position information of the target segmented word in the training sample and the connection relationship between the target segmented word and other words.
[0055] It should be noted that in this embodiment, the position feature vector is a feature vector representing the specific position information of the target segmented word in the training sample, and the chunk feature vector is a feature vector representing the segmentation standard of the target segmented word and the connection relationship between the target segmented word and other words. After determining the target segmented word, look up the target segmented word in the preset word vector table, and use the word feature vector corresponding to the target segmented word in the preset word vector table as the word feature vector of the target segmented word.
[0056] In one embodiment, the semantic feature vector represents a feature vector divided according to the units of meaning in language, where semantic elements are semantic components or semantic features, including semantic phrases, semantic clusters, and semantic positions. Taking target word segmentation as an example, obtaining the semantic feature vector of each word segment can include: according to the mapping relationship between words and semantic feature vectors in a preset semantic feature vector table, searching for the semantic feature vector of each word segment in the preset semantic feature vector table.
[0057] S104. Based on the occlusion task and sentence relationship task in the pre-trained language model, the input text vector is processed to obtain the loss function of the pre-trained language model, and the pre-trained language model is trained based on the loss function.
[0058] In this embodiment, the pre-trained language model is a self-supervised learning model that runs on a massive amount of training samples, capable of determining an accurate feature representation for words. The types of pre-trained language models can be: BERT, Mengzi, CPM, or mT5.
[0059] In this embodiment, the loss function is a function that maps the values of a random event or its related random variables to non-negative real numbers to represent the "risk" or "loss" of that random event. In applications, the loss function is often used as a learning criterion in conjunction with optimization problems, i.e., solving and evaluating pre-trained language models by minimizing the loss function.
[0060] In this embodiment, the occlusion task can be Masked Language Modeling (MLM). Specifically, the occlusion task is a self-supervised learning problem and is usually treated as a classification problem. The occlusion task can obscure a single character, a word, or punctuation marks. Specifically, in the training samples, 15% of the characters or words are randomly selected as the objects to be obscured. Then, 80% of these obscured objects are selected for obscuration, 10% are selected for replacement, and 10% are left untouched. In this embodiment, the sentence relationship task describes the correlation between sentences. For example, the sentence relationship task can include sentence equivalence tasks, sentence expansion tasks, or sentence reduction tasks. After processing the input text vector through occlusion and sentence relationship tasks, the loss function of the pre-trained language model can be obtained. The loss function is compared with a preset value. If the current loss function value is greater than the preset value, S101 to S104 are repeated until the loss function is less than the preset value, thus completing the training of the pre-trained language model based on the loss function.
[0061] For a better illustration of this application, please refer to [link to relevant documentation]. Figure 2aThe diagram shows the architecture of a pre-trained language model. It can be seen that the input text vector undergoes occlusion and sentence relationship tasks, which ultimately determine the loss function of the pre-trained language model.
[0062] This application provides a training method for a pre-trained language model. The method involves acquiring training samples and corresponding sequence text information, where the training samples are obtained from an original training sample library. The sequence text information is segmented to obtain multiple segments within the training samples. The basic feature vector and semantic feature vector of each segment are obtained, and these are concatenated to obtain an input text vector. Based on the occlusion and sentence relationship tasks in the pre-trained language model, the input text vector is processed to obtain a loss function for the pre-trained language model, which is then used to train the model. This technical solution improves the semantic representation of each segment by acquiring its semantic feature vector. Generating the input text vector from the segmented words further enhances its semantic representation. Training the pre-trained language model with this input text vector improves its semantic representation capability, enabling it to accurately predict the sentiment classification of text and thus achieve user complaint tendency identification based on text sentiment.
[0063] Figure 2b This is a flowchart illustrating a training method for a pre-trained language model according to Embodiment 2 of this application. Embodiment 2 includes the following steps:
[0064] S201. Obtain the training samples and the corresponding sequence text information.
[0065] For example, this step can refer to step S101 above, and will not be repeated here.
[0066] S202. Perform word segmentation on the sequence text information to obtain multiple words included in the training samples.
[0067] For example, this step can refer to step S102 above, and will not be repeated here.
[0068] S203. Obtain the basic feature vector of each word in multiple word segments, and obtain the semantic feature vector of each word segment. Then, concatenate the basic feature vector of each word segment with the semantic feature vector of each word segment to obtain the input text vector.
[0069] For example, this step can refer to step S103 above, and will not be repeated here.
[0070] S204. Based on the occlusion task in the pre-trained language model, the input text vector is processed to obtain the loss function of the occlusion task.
[0071] In this embodiment, after the input text vector is processed by the occlusion task in the pre-trained language model, the occluded words are predicted based on the semantic features of the input text vector. The text vector generated by the predicted occluded words is then compared with the text vector generated by the actual occluded words to calculate the similarity. If the value of the similarity calculation result is within a preset range, then the loss function at this time is determined to be the loss function of the occlusion task.
[0072] S205. Based on the sentence relation task in the pre-trained language model, the input text vector is processed to obtain the loss function of the sentence relation task.
[0073] Statement relationship tasks include: statement equivalence tasks, statement expansion tasks, and statement reduction tasks. Statement equivalence tasks involve statements that express similar meanings. Statement expansion tasks involve extending the meaning of one statement to the meaning of another. Statement reduction tasks involve reducing the meaning of one statement to the meaning of another. For example, the two statements in a statement equivalence task could be "I am a teacher" and "The teacher is me." The two statements in a statement expansion task could be "I am a teacher" and "I am a teacher at Chengdu No. 7 Middle School." The two statements in a statement reduction task could be "I am a Chinese language teacher at Chengdu No. 7 Middle School" and "I am a teacher."
[0074] In this embodiment, the input text vector is processed based on the sentence relation task in the pre-trained language model to obtain the loss function of the sentence relation task. This includes: processing the input text vector based on the sentence relation task in the pre-trained language model to obtain a predicted value; and determining the loss function of the sentence relation task based on the distance between the predicted value and the true value. Wherein, the true value represents the actual sentence relation of the input text vector; and the predicted value represents the sentence relation of the input text vector predicted by the pre-trained language model.
[0075] The input text vector is obtained by processing training samples; the training samples are determined based on the original training sample library; the training samples include positive samples and negative samples; the training samples for the positive samples are determined by the original samples according to the sentence relationship rules, and the original samples are selected from the original training sample library; the training samples for the negative samples are determined from the original training sample library excluding the training samples used to generate the positive samples.
[0076] In this embodiment, the distance between the predicted value and the true value represents the similarity between the predicted value and the true value. The specific method for calculating the distance value can be the cosine similarity method.
[0077] In this embodiment, the input text vector is input into a pre-trained language model to obtain the predicted value of the sentence relationship of the input text vector. By comparing the predicted value with the true value, the loss function of the sentence relationship task is determined.
[0078] S206. Determine the loss function of the pre-trained language model based on the loss function of the occlusion task and the loss function of the sentence relationship task.
[0079] In this embodiment, the loss function of the pre-trained language model is determined by jointly using the loss function of the occlusion task and the loss function of the sentence relationship task, which can make the accuracy of the pre-trained language model determined based on the loss function of the pre-trained language model higher.
[0080] In this embodiment, the loss function of the pre-trained language model is determined based on the loss function of the occlusion task and the loss function of the sentence relationship task, including: calculating the sum of the loss functions of the occlusion task and the loss functions of the sentence relationship task; and determining the loss function of the pre-trained language model based on the difference between the sum and a preset value.
[0081] In one embodiment, the pre-trained language model can be connected to an output layer as a pre-trained model to form a sentiment recognition model. This sentiment recognition model is used to identify the sentiment category reflected in a piece of text. Sentiment categories include positive sentiment, neutral sentiment, and negative sentiment.
[0082] Specifically, the pre-trained language model is concatenated with the output layer to obtain the sentiment recognition model; training data is acquired, including historical comment texts in the agent service scenario and the sentiment categories reflected in the historical comment texts; the sentiment recognition model is trained based on the training data; wherein, the trained sentiment recognition model is used to identify the sentiment category reflected in any text.
[0083] This application provides a training method for a pre-trained language model. Based on the occlusion task in the pre-trained language model, the input text vector is processed to obtain a loss function for the occlusion task. Based on the sentence relationship task in the pre-trained language model, the input text vector is processed to obtain a loss function for the sentence relationship task. Based on the loss functions of the occlusion task and the sentence relationship task, the loss function of the pre-trained language model is determined. The pre-trained language model is then trained based on these loss functions. In this technical solution, the loss function of the pre-trained language model is jointly determined by the loss functions of the occlusion task and the sentence relationship task. The loss function of the occlusion task can train the pre-trained language model's ability to predict words, while the loss function of the sentence relationship task can train the pre-trained language model's ability to predict relationships between sentences. Therefore, it can improve the accuracy of the trained pre-trained language model in recognizing text. Furthermore, since the trained pre-trained language model can be used as a pre-trained model and connected to an output layer to form a sentiment recognition model, this sentiment recognition model is used to identify the sentiment category reflected in a text. This further improves the accuracy of the sentiment recognition model in identifying the sentiment category of text.
[0084] Figure 3 This is a flowchart illustrating a text sentiment classification method according to Embodiment 3 of this application. Embodiment 3 includes the following steps:
[0085] S301. Obtain the comment text to be classified into sentiment and the corresponding sequence text information.
[0086] For example, the review text to be categorized by sentiment can be obtained from review applications. This review text can be user experiences or collected user reviews of preset products, which can be unreleased products or products already used by users. The review text to be categorized by sentiment can also be collected user reviews of telephone agents.
[0087] S302. Perform word segmentation on the sequence text information to obtain multiple words included in the comment text.
[0088] For example, this step can refer to step S102 above, and will not be repeated here.
[0089] S303. Obtain the basic feature vector of each word in multiple word segments, and obtain the semantic feature vector of each word segment. Then, concatenate the basic feature vector of each word segment and the semantic feature vector of each word segment to obtain the input text vector.
[0090] For example, this step can refer to step S103 above, and will not be repeated here.
[0091] S304. Input the input text vector into the sentiment recognition model for sentiment recognition to obtain the sentiment category reflected by the comment text; wherein, the sentiment recognition model includes: a pre-trained language model and an output layer; the pre-trained language model is used to determine the sentiment category of the comment text based on the occlusion task and sentence relationship task in the sentiment recognition model, and the output layer is used to output the sentiment category; the pre-trained language model is trained by the method in Example 1 or Example 2.
[0092] For example, the input text vector is generated from the review text to be classified for sentiment. After inputting the input text vector into the sentiment recognition model, the sentiment category reflected by the review text can be predicted. Then, the user's sentiment category towards a preset product can be determined, and the preset product can be adjusted according to the user's preferences to better suit the user's usage habits. For products not yet on the market, this allows the product to better suit the usage habits of most users after its launch.
[0093] If the input text vector is generated from user comments on telephone agents, feeding the input text vector into the sentiment recognition model can predict the sentiment category reflected in the comment text. Then, the service process of telephone agents can be adjusted to improve the user experience.
[0094] This application provides a text sentiment classification method that, after identifying the sentiment category of the comment text to be classified, can adjust the product evaluated by the comment text based on the sentiment classification results, thereby enabling evidence-based product adjustments and improving user experience.
[0095] Figure 4 This is a schematic diagram of a training device for a pre-trained language model according to Embodiment 4 of this application. The device 40 in Embodiment 4 includes:
[0096] The acquisition unit 401 is used to acquire training samples and corresponding sequence text information; wherein, the training samples are acquired from the original training sample library.
[0097] The processing unit 402 is used to perform word segmentation on the sequence text information to obtain multiple words included in the training samples.
[0098] The acquisition unit 401 is also used to acquire the basic feature vector of each word in multiple word segments, and to acquire the semantic feature vector of each word segment;
[0099] The concatenation unit 403 is used to concatenate the basic feature vector of each word and the semantic feature vector of each word to obtain the input text vector.
[0100] The processing unit 402 is also used to process the input text vector based on the occlusion task and the sentence relationship task in the pre-trained language model, respectively, to obtain the loss function of the pre-trained language model, and to train the pre-trained language model based on the loss function.
[0101] In one example, the basic feature vector of each word segment includes a position feature vector, a phrase feature vector, and a word feature vector; among multiple word segments is a target word, and when the acquisition unit 401 acquires the basic feature vector of the target word, it performs the following steps:
[0102] Based on the mapping relationship between words and word feature vectors in the pre-defined word vector table, the word feature vector corresponding to the target word is searched in the pre-defined word vector table. Based on the position information of the target word in the training samples and the connection relationship between the target word and other words, the position feature vector and the chunk feature vector of the target word are determined.
[0103] In one example, when acquiring the semantic feature vector of each word segment, the acquisition unit 401 performs the following steps:
[0104] Based on the mapping relationship between words and semantic feature vectors in the preset semantic feature vector table, the semantic feature vector of each word segment is searched in the preset semantic feature vector table.
[0105] In one example, when processing unit 402 processes the input text vector based on the occlusion task and sentence relationship task in the pre-trained language model and obtains the loss function of the pre-trained language model, it performs the following steps:
[0106] Based on the occlusion task in the pre-trained language model, the input text vector is processed to obtain the loss function for the occlusion task; based on the sentence relation task in the pre-trained language model, the input text vector is processed to obtain the loss function for the sentence relation task; based on the loss functions of the occlusion task and the sentence relation task, the loss function of the pre-trained language model is determined.
[0107] In one example, when processing unit 402 processes the input text vector to obtain the loss function for the sentence relation task based on the pre-trained language model, it performs the following steps:
[0108] Based on the sentence relation task in the pre-trained language model, the input text vector is processed to obtain the predicted value;
[0109] The loss function for the sentence relationship task is determined based on the distance between the predicted and true values; where the true value represents the actual sentence relationship of the input text vector, and the predicted value represents the sentence relationship of the input text vector predicted by the pre-trained language model.
[0110] The input text vector is obtained by processing the training samples; the training samples include positive samples and negative samples; the training samples for positive samples are determined from the training samples according to the sentence relationship rules; and the training samples for negative samples are determined from the original training sample library after removing the training samples used to generate positive samples.
[0111] In one example, statement relation tasks include: statement equivalence tasks, statement expansion tasks, or statement reduction tasks.
[0112] In one example, when the processing unit 402 determines the loss function of the pre-trained language model based on the loss function of the occlusion task and the loss function of the sentence relationship task, it performs the following steps:
[0113] Calculate the sum of the loss functions for the occlusion task and the statement relationship task;
[0114] The loss function of the pre-trained language model is determined based on the difference between the sum and the preset value.
[0115] In one example, the device also includes:
[0116] Unit 404 is used to concatenate the pre-trained language model with the output layer to obtain the emotion recognition model.
[0117] The acquisition unit 401 is also used to acquire training data, which includes historical comment texts in the agent service scenario and the sentiment categories reflected by the historical comment texts.
[0118] Training unit 405 is used to train the emotion recognition model based on training data; wherein, the trained emotion recognition model is used to identify the emotion category reflected in any text.
[0119] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0120] Figure 5 This is a schematic diagram of a text sentiment classification device according to Embodiment Six of this application. The device 50 in Embodiment Six includes:
[0121] The acquisition unit 501 is used to acquire the comment text to be classified into sentiment and the corresponding sequence text information.
[0122] The processing unit 502 is used to perform word segmentation on the sequence text information to obtain multiple words included in the comment text.
[0123] The concatenation unit 503 is used to obtain the basic feature vector of each word in multiple word segments, as well as the semantic feature vector of each word segment, and concatenate the basic feature vector and the semantic feature vector of each word segment to obtain the input text vector.
[0124] The determining unit 504 is used to input the input text vector into the sentiment recognition model for sentiment recognition to obtain the sentiment category reflected by the comment; wherein, the sentiment recognition model includes: a pre-trained language model and an output layer; the pre-trained language model is used to determine the sentiment category of the comment text based on the occlusion task and sentence relationship task in the sentiment recognition model, and the output layer is used to output the sentiment category; the pre-trained language model is trained by the method in the first aspect.
[0125] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0126] Figure 6 This is a block diagram illustrating an electronic device according to an exemplary embodiment. The device may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc.
[0127] Electronic device 600 may include one or more of the following components: processing component 602, memory 604, power supply component 606, multimedia component 608, audio component 610, input / output (I / O) interface 612, sensor component 614, and communication component 616.
[0128] Processing component 602 typically controls the overall operation of electronic device 600, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 602 may include one or more processors 620 to execute instructions to complete all or part of the steps of the pre-trained language model training method described above, or to complete all or part of the steps of the text sentiment classification method described above. Furthermore, processing component 602 may include one or more modules to facilitate interaction between processing component 602 and other components. For example, processing component 602 may include a multimedia module to facilitate interaction between multimedia component 608 and processing component 602.
[0129] Memory 604 is configured to store various types of data to support the operation of electronic device 600. Examples of such data include instructions for any application or method operating on electronic device 600, contact data, phonebook data, messages, pictures, videos, etc. Memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.
[0130] Power supply component 606 provides power to various components of electronic device 600. Power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 600.
[0131] Multimedia component 608 includes a screen that provides an output interface between electronic device 600 and user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of a touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 608 includes a front-facing camera and / or a rear-facing camera. When electronic device 600 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.
[0132] Audio component 610 is configured to output and / or input audio signals. For example, audio component 610 includes a microphone (MIC) configured to receive external audio signals when electronic device 600 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 604 or transmitted via communication component 616. In some embodiments, audio component 610 also includes a speaker for outputting audio signals.
[0133] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, start buttons, and lock buttons.
[0134] Sensor assembly 614 includes one or more sensors for providing state assessments of various aspects of electronic device 600. For example, sensor assembly 614 can detect the on / off state of electronic device 600, the relative positioning of components such as the display and keypad of electronic device 600, changes in position of electronic device 600 or a component of electronic device 600, the presence or absence of user contact with electronic device 600, orientation or acceleration / deceleration of electronic device 600, and temperature changes of electronic device 600. Sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 614 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 614 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.
[0135] Communication component 616 is configured to facilitate wired or wireless communication between electronic device 600 and other devices. Electronic device 600 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 616 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 616 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
[0136] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.
[0137] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, which can be executed by a processor 620 of an electronic device 600 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.
[0138] A non-transitory computer-readable storage medium, wherein instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the training method of the pre-trained language model or the text sentiment classification method described above.
[0139] This application also discloses a computer program product, including a computer program that, when executed by a processor, implements the method described in this embodiment.
[0140] Various embodiments of the systems and technologies described above in this application can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0141] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or electronic device.
[0142] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0143] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0144] The systems and technologies described herein can be implemented in computing systems that include back-end components (e.g., as data electronic devices), or computing systems that include middleware components (e.g., application electronic devices), or computing systems that include front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.
[0145] Computer systems can include client and electronic devices. Clients and electronic devices are generally geographically separated and typically interact via communication networks. The client-electronic device relationship is created by computer programs running on the respective computers and having a client-electronic device relationship with each other. The electronic device can be a cloud electronic device, also known as a cloud computing electronic device or cloud host, a host product within the cloud computing service system, addressing the shortcomings of traditional physical hosts and VPS services ("Virtual Private Server," or simply "VPS") in terms of management difficulty and weak business scalability. The electronic device can also be an electronic device in a distributed system or an electronic device incorporating blockchain technology. It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application is achieved, and this is not limited herein.
[0146] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.
[0147] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.
Claims
1. A training method for a pre-trained language model, characterized in that, include: Obtain training samples and the corresponding sequence text information of the training samples; The sequence text information is segmented to obtain multiple segments including the training sample, and the multiple segments include the target segment. The basic feature vector of each word in the plurality of word segments is obtained, as well as the semantic feature vector of each word segment. The basic feature vector and the semantic feature vector of each word segment are concatenated to obtain the input text vector. The basic feature vector of each word segment includes a position feature vector, a chunk feature vector, and a word feature vector. The chunk feature vector is used to characterize the segmentation criteria of the target word segment and the connection relationship between the target word segment and other words. Based on the occlusion task and sentence relationship task in the pre-trained language model, the input text vector is processed to obtain the loss function of the pre-trained language model, and the pre-trained language model is trained based on the loss function.
2. The method according to claim 1, characterized in that, Obtain the basic feature vector of the target word segmentation, including: Based on the mapping relationship between words and word feature vectors in the preset word vector table, the word feature vector corresponding to the target word segmentation is searched in the preset word vector table; Based on the position information of the target word in the training samples and the connection relationship between the target word and other words, the position feature vector and the chunk feature vector of the target word are determined.
3. The method according to claim 1, characterized in that, The process of obtaining the semantic feature vector for each word segment includes: Based on the mapping relationship between words and semantic feature vectors in the preset semantic feature vector table, the semantic feature vector of each word segment is searched in the preset semantic feature vector table.
4. The method according to claim 1, characterized in that, The occlusion task and sentence relationship task based on the pre-trained language model are used to process the input text vector to obtain the loss function of the pre-trained language model, including: Based on the occlusion task in the pre-trained language model, the input text vector is processed to obtain the loss function for the occlusion task; Based on the sentence relation task in the pre-trained language model, the input text vector is processed to obtain the loss function of the sentence relation task; The loss function of the pre-trained language model is determined based on the loss function of the occlusion task and the loss function of the sentence relationship task.
5. The method according to claim 4, characterized in that, The sentence relation task based on the pre-trained language model processes the input text vector to obtain the loss function for the sentence relation task, including: Based on the sentence relation task in the pre-trained language model, the input text vector is processed to obtain the predicted value; The loss function for the sentence relationship task is determined based on the distance between the predicted value and the true value; wherein the true value represents the actual sentence relationship of the input text vector; and the predicted value represents the sentence relationship of the input text vector predicted by the pre-trained language model. The input text vector is obtained by processing training samples; the training samples are determined based on the original training sample library; the training samples include positive samples and negative samples; the positive samples are determined by the original samples according to the sentence relationship rules, and the original samples are selected from the original training sample library; the negative samples are determined from the original training sample library after removing the training samples used to generate the positive samples.
6. The method according to claim 5, characterized in that, The statement relation task includes: statement equivalence task, Statement expansion task or statement reduction task.
7. The method according to claim 4, characterized in that, The step of determining the loss function of the pre-trained language model based on the loss function of the occlusion task and the loss function of the sentence relationship task includes: Calculate the sum of the loss function for the occlusion task and the loss function for the statement relationship task; The loss function of the pre-trained language model is determined based on the difference between the sum and the preset value.
8. The method according to claim 1, characterized in that, The method further includes: The trained pre-trained language model is concatenated with the output layer to obtain the emotion recognition model; Acquire training data, which includes historical comment texts in the agent service scenario and the sentiment categories reflected by the historical comment texts; The emotion recognition model is trained based on the training data; wherein, the trained emotion recognition model is used to identify the emotion category reflected in any text.
9. A text sentiment classification method, characterized in that, include: Obtain the comment text to be classified into sentiment and the corresponding sequence text information of the comment text; The sequence text information is segmented to obtain multiple segments of the comment text, including the target segment; The basic feature vector of each word in the plurality of word segments is obtained, as well as the semantic feature vector of each word segment. The basic feature vector and the semantic feature vector of each word segment are concatenated to obtain the input text vector. The basic feature vector of each word segment includes a position feature vector, a chunk feature vector, and a word feature vector. The chunk feature vector is used to characterize the segmentation criteria of the target word segment and the connection relationship between the target word segment and other words. The input text vector is input into a sentiment recognition model for sentiment recognition to obtain the sentiment category reflected by the comment text; wherein, the sentiment recognition model includes: a pre-trained language model and an output layer; the pre-trained language model is used to determine the sentiment category of the comment text based on the occlusion task and the sentence relationship task in the sentiment recognition model, and the output layer is used to output the sentiment category; the pre-trained language model is trained by the method described in any one of claims 1-8.
10. A training device for a pre-trained language model, characterized in that, The device includes: The acquisition unit is used to acquire training samples and the corresponding sequence text information of the training samples; The processing unit is used to perform word segmentation processing on the sequence text information to obtain multiple word segments included in the training sample, wherein the multiple word segments include the target word; The acquisition unit is also used to acquire the basic feature vector of each word in the plurality of word segments, and to acquire the semantic feature vector of each word segment; The concatenation unit is used to concatenate the basic feature vector of each word segment and the semantic feature vector of each word segment to obtain the input text vector; the basic feature vector of each word segment includes a position feature vector, a chunk feature vector, and a word feature vector; the chunk feature vector is used to characterize the segmentation criteria of the target word segment and the connection relationship between the target word segment and other words; The processing unit is further configured to process the input text vector based on the occlusion task and the sentence relationship task in the pre-trained language model, respectively, to obtain the loss function of the pre-trained language model, and to train the pre-trained language model based on the loss function.
11. A text sentiment classification device, characterized in that, The device includes: The acquisition unit is used to acquire the comment text to be classified into sentiment and the corresponding sequence text information of the comment text; The processing unit is used to perform word segmentation processing on the sequence text information to obtain multiple words included in the comment text, wherein the multiple words include the target word; The concatenation unit is used to obtain the basic feature vector of each word in the plurality of word segments, and to obtain the semantic feature vector of each word segment. The basic feature vector of each word segment and the semantic feature vector of each word segment are concatenated to obtain the input text vector. The basic feature vector of each word segment includes a position feature vector, a chunk feature vector, and a word feature vector. The chunk feature vector is used to characterize the segmentation criteria of the target word segment and the connection relationship between the target word segment and other words. A determining unit is configured to input the input text vector into a sentiment recognition model for sentiment recognition to obtain the sentiment category reflected by the comment; wherein, the sentiment recognition model includes: a pre-trained language model and an output layer; the pre-trained language model is used to determine the sentiment category of the comment text based on the occlusion task and the sentence relationship task in the sentiment recognition model, and the output layer is used to output the sentiment category; the pre-trained language model is trained by the method of any one of claims 1-8.
12. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-8 or claim 9.
13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-8 or 9.